Information processing device and information processing method

Abstract

[Problem] To provide technology capable of reconstructing a model accurately in three dimensions. [Solution] An information processing device according to the present technology is equipped with a control unit. The control unit generates a plurality of interpolation points for interpolating between a plurality of points in the height direction on the basis of a plurality of first and second points among said plurality of points, each of which includes information regarding a coordinate position in three dimensions and is used for three-dimensional reconstruction of a model, said plurality of first points having a height coordinate within a first range and said plurality of second points having a height coordinate within a second range lower than the first range.

Description

Information processing device and information processing method

[0001] The present technology relates to processing three-dimensional point clouds used for three-dimensional reconstruction of models.

[0002] In recent years, techniques for three-dimensionally reconstructing a model from a three-dimensional point cloud, such as the Poisson mesh reconstruction method and the Delaunay mesh reconstruction method, have become known.

[0003] For example, in Patent Document 1 below, a three-dimensional point cloud is generated from two two-dimensional images of the same scene captured from different viewpoints. Then, a plurality of triangular meshes that form the outline of a model are generated from the three-dimensional point cloud by Delaunay mesh reconstruction, and texture information extracted from the two-dimensional images is combined with the triangular meshes. This results in a three-dimensional reconstruction of the model.

[0004] Special Publication No. 2009-536499

[0005] In such fields, there is a demand for technology that enables accurate three-dimensional reconstruction of models.

[0006] In view of the above circumstances, an object of the present technology is to provide a technology that enables accurate three-dimensional reconstruction of a model.

[0007] An information processing device according to the present technology includes a control unit, and the control unit generates a plurality of interpolation points for interpolating the plurality of points in a height direction based on a plurality of first points whose height coordinates are within a first range and a plurality of second points whose height coordinates are within a second range lower than the first range, among a plurality of points used for three-dimensional reconstruction of a model, each of which includes information on a three-dimensional coordinate position.

[0008] In this way, by generating a plurality of new interpolation points, it is possible to accurately reconstruct a three-dimensional model.

[0009] According to another aspect of the present technology, there is provided an information processing device including a control unit, the control unit including information on three-dimensional coordinate positions and information on normal vectors, and the control unit correcting the normal vector of a predetermined point among a plurality of first points used for three-dimensional reconstruction of a model, based on a plurality of first points whose height coordinates are within a first range and a plurality of second points whose height coordinates are within a second range lower than the first range.

[0010] In this way, by correcting the normal vector of a predetermined point, it is possible to accurately reconstruct the model in three dimensions.

[0011] The information processing method according to the present technology generates a plurality of interpolation points for interpolating the plurality of points in the height direction based on a plurality of first points whose height coordinates are within a first range and a plurality of second points whose height coordinates are within a second range lower than the first range, each of which includes information on three-dimensional coordinate positions and is used for three-dimensional reconstruction of a model.

[0012] 13 is a diagram showing an information processing device according to a first embodiment of the present technology. FIG. 14 is a diagram showing two aerial images of the same building captured from slightly different viewpoints. FIG. 15 is a diagram showing a state when a model is three-dimensionally reconstructed from a three-dimensional point cloud generated based on the aerial image shown in FIG. 2. FIG. 16 is a diagram showing a state when a mesh is generated by a three-dimensional reconstruction method based on coordinate information and normal vector information at each point in the three-dimensional point cloud. FIG. 17 is a diagram showing a state when a mesh is generated based on a three-dimensional point cloud that has been subjected to interpolation point generation processing and normal vector correction processing in the present technology. FIG. 18 is a flowchart showing processing by a control unit according to the present embodiment. FIG. 19 is a flowchart showing processing by a control unit according to the present embodiment. FIG. 19 is a diagram showing a state when region division processing is executed. FIG. 19 is a diagram showing three-dimensional divided regions whose heights are equal to or greater than a predetermined threshold and three-dimensional divided regions whose heights are less than the predetermined threshold. FIG. 20 is a diagram showing an example when the number of points included in a three-dimensional divided region is small. FIG. 21 is a diagram showing three-dimensional divided regions and an enclosed region. FIG. 22 is a diagram for explaining normal vector correction processing and interpolation point generation processing. FIG. 23 is a perspective view of a building, showing normal correction points whose normal vectors have been corrected, and interpolation points. FIG. 24 is a diagram showing a normal map corresponding to FIG. 13. FIG. 25 is a top view of a building, showing normal correction points. FIG. 1 is a top view showing a terrain, and is a diagram showing normal correction points. FIG. 2 is a diagram showing a state when a building is three-dimensionally reconstructed by the Poisson mesh reconstruction method (meshing). FIG. 3 is a diagram showing an orthoimage. FIG. 4 is a diagram for explaining the operation etc. of this embodiment. FIG. 5 is another diagram for explaining the operation etc. of this embodiment.

[0013] Hereinafter, embodiments of the present technology will be described with reference to the drawings.

[0014] First Embodiment Overall Configuration and Configuration of Each Unit FIG. 1 is a diagram showing an information processing device 100 according to a first embodiment of the present technology.

[0015] In this embodiment, the information processing device 100 is, for example, a general-purpose computer such as a personal computer (PC) (desktop type, laptop type, tablet type, etc.), a smartphone, etc. Note that the information processing device 100 may be a device dedicated to the present technology.

[0016] As shown in FIG. 1, the information processing device 100 includes a control unit 21 , a storage unit 22 , a display unit 23 , an input unit 24 , and a communication unit 25 .

[0017] The control unit 21 performs overall control of the information processing device 100 based on various programs stored in the storage unit 22. In this embodiment, the storage unit 22 stores programs related to the present technology, and the control unit 21 executes various processes described in the operation description section below in accordance with these programs. Specific processes performed by the control unit 21 will be described in detail later.

[0018] The control unit 21 is realized by hardware or a combination of hardware and software. The hardware is configured as part or all of the control unit 21, and examples of this hardware include a central processing unit (CPU), a graphics processing unit (GPU), a vision processing unit (VPU), a neural processing unit (NPU), a digital signal processor (DSP), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SOC), or a combination of two or more of these.

[0019] The storage unit 22 includes a non-volatile memory that stores various programs and various data required for processing by the control unit 21, and a volatile memory that is used as a work area for the control unit 21.

[0020] The various programs described above may be read from a portable recording medium such as an optical disk or semiconductor memory, or may be downloaded from a server device on a network.

[0021] The display unit 23 is, for example, a liquid crystal display, and displays various images on the screen in response to instructions from the control unit 21 .

[0022] The input unit 24 is, for example, a mechanical push button or a contact sensor provided on the display unit 23 , and detects input by the user and outputs it to the control unit 21 .

[0023] The communication unit 25 is configured to be able to communicate with other devices via wire or wirelessly.

[0024] <Basic Concept of the Present Technology> Next, the basic concept of the present technology will be described. First, consider a case where a point cloud PG for 3D reconstruction of a model is generated from a plurality of aerial images taken of the same location from different viewpoints using a drone or the like, using SfM / MVS technology (SfM: Structure from Motion, MVS: Multi-View Stereo) or the like.

[0025] Figure 2 shows two aerial images of the same building taken from slightly different viewpoints. In the aerial image on the left in Figure 2, a small portion of the building wall is visible, while in the aerial image on the right, the building wall is not visible.

[0026] Therefore, for example, when attempting to generate a three-dimensional point group PG (multiple points P in three dimensions) from an image such as that shown in Figure 2, it may not be possible to properly generate a three-dimensional point group PG corresponding to the wall surface of a building.

[0027] Figure 3 is a diagram showing how a model is 3D reconstructed from a 3D point cloud PG generated based on the aerial image shown in Figure 2. As shown at the top of Figure 3, when a 3D point cloud PG is generated based on the aerial image shown in Figure 2, an error occurs in which a 3D point cloud PG corresponding to the building's wall surfaces cannot be generated, and the areas corresponding to the wall surfaces are left blank. If a building (model) is 3D reconstructed (meshed) based on such a 3D point cloud PG, the shape of the building will be distorted at the positions corresponding to the building's wall surfaces, as shown in the second and bottom diagrams from the bottom in Figure 3. In other words, the model cannot be 3D reconstructed accurately.

[0028] Therefore, in this technology, for a location that is only slightly visible in the image, such as a location corresponding to the wall of a building, a plurality of interpolation points IP are generated to interpolate the 3D point cloud PG, thereby enabling accurate 3D reconstruction of the model.

[0029] Here, each point P in the three-dimensional point group PG includes three-dimensional coordinate information. Furthermore, each point P in the three-dimensional point group PG may also include normal vector information. The three-dimensional coordinate information and normal vector information are generated based on a plurality of images such as those shown in FIG. 2 . Note that the normal vector information is used, for example, in meshing processing in the Poisson mesh reconstruction method.

[0030] FIG. 4 is a diagram showing how a mesh is generated by a three-dimensional reconstruction method based on coordinate information and normal vector information at each point P in a three-dimensional point group PG.

[0031] First, on the left side of Fig. 4, black dots represent points P in the 3D point group PG, and arrows represent normal vectors. Also, on the left side of Fig. 4, one cube represents one area for generating one mesh (the same applies to the right side of Fig. 4).

[0032] On the left side of Fig. 4, the two points P included in the upper left cube are set to be points P corresponding to the roof of the building, and the three points P included in the upper right cube are set to be points P corresponding to the corner where the roof and wall of the building join. Also, on the left side of Fig. 4, the one point P included in the second cube from the bottom is set to be point P corresponding to the underside of the wall of the building, and the three points P included in the bottom cube are set to be points P corresponding to the ground.

[0033] As mentioned above, the image from which the normal vectors are generated barely shows the building's walls. Therefore, on the left side of Figure 4, the directions of the normal vectors of the three points P in the upper right cube that correspond to the roof and corners of the building's walls, and the direction of the normal vector of the one point P in the second cube from the bottom that corresponds to the lower side of the wall, tend to be inaccurate. In such cases, the direction of the normal vectors tends to point diagonally upward.

[0034] Therefore, if a mesh is generated based on an inaccurate normal vector as shown on the left side of Figure 4, the orientation of the mesh will be inaccurate, as shown on the right side of Figure 4, and the model cannot be accurately reconstructed in 3D (see also the second from the bottom and the bottom of Figure 3).

[0035] Therefore, in this technology, a process of correcting the normal vector is performed for a predetermined point P (e.g., a point P corresponding to a corner of a roof or wall of a building) in the three-dimensional point cloud PG. Typically, the direction of the normal vector is set to a direction in a plane (xy plane direction). This allows the technology to more accurately reconstruct the model in three dimensions. Note that in the description herein, the point P in the three-dimensional point cloud PG whose normal vector is corrected is referred to as a normal correction point RP.

[0036] FIG. 5 is a diagram showing how a mesh is generated based on a three-dimensional point group PG that has been subjected to the interpolation point generation process and the normal vector correction process in this technology.

[0037] On the left side of Fig. 5, white circles indicate interpolation points IP, and gray circles indicate normal correction points RP. Note that on the left side of Fig. 5, black circles indicate points P that are treated as original points (points P that are neither interpolation points IP nor normal correction points RP).

[0038] As shown on the left side of Figure 5, in this technology, interpolation points IP are generated at predetermined intervals along the height direction at locations corresponding to, for example, the wall surfaces of a building. Furthermore, the normals of points P in the three-dimensional point cloud PG, for example, those corresponding to corners where the roof and wall surfaces of the building join, are corrected. Note that the newly generated interpolation points IP do not originally contain normal vector information, so new normal vector information is generated for these interpolation points IP.

[0039] 5, when a mesh is generated based on the 3D point group PG that has been subjected to the interpolation point generation process, the normal vector correction process, etc., it is possible to accurately generate a mesh that corresponds to, for example, a wall surface. Therefore, with this technology, it is possible to accurately reconstruct a model in 3D.

[0040] The above is the basic concept of the present technology. Note that the interpolation point generation process, normal vector correction process, etc. will be described in detail in the section on operation below.

[0041] <Explanation of Operation> Next, the processing of the control unit 21 according to this embodiment will be described in detail. Figures 6 and 7 are flowcharts showing the processing of the control unit 21 according to this embodiment.

[0042] First, the control unit 21 generates a three-dimensional point cloud PG including a plurality of three-dimensional points P (ST101). In ST101, the control unit 21 first extracts a plurality of points P as feature points from a plurality of two-dimensional images (e.g., see FIG. 2 ) captured of the same location from different viewpoints. Then, the control unit 21 generates three-dimensional coordinate information and normal vector information for each of the plurality of points P extracted as feature points (the three-dimensional point cloud PG) based on the two-dimensional images. Then, the control unit 21 associates this three-dimensional coordinate information and normal vector information with the corresponding point P and stores them in the storage unit 22.

[0043] In this embodiment, the three-dimensional coordinates of each point P in the three-dimensional point group PG are expressed as (x, y, z) values ​​in a three-dimensional Cartesian coordinate system (the same applies to the coordinate system representing the normal vector). Note that the three-dimensional coordinates may be expressed as (r, θ, φ) or the like in spherical coordinates, and any coordinate system that can represent a point in three-dimensional space may be used (the same applies to the coordinate system representing the normal vector).

[0044] As a process for generating a three-dimensional point cloud PG, for example, SfM / MVS technology is used, but typically, any technology that can generate a three-dimensional point cloud PG (and can generate normal vectors) can be used.

[0045] In addition, in the description of this embodiment, for ease of understanding, the 2D images from which the 3D point cloud PG is generated will be described using an aerial image captured by a drone or the like as an example (see FIG. 2). Note that applying the present technology to such aerial images is particularly advantageous from the perspective of accurate 3D reconstruction of a model, but the 2D images are not limited to aerial images. For example, the 2D images may be either outdoor or indoor images, and there are no particular limitations on the captured object (the model to be 3D reconstructed).

[0046] The three-dimensional point cloud PG generated in ST101 does not include a point cloud PG corresponding to the wall surfaces of the building (see the top of FIG. 3).

[0047] After generating the 3D point cloud PG, the control unit 21 next determines whether the user has input an instruction to perform high-quality processing for the 3D reconstruction of the model (ST102). If the user has input an instruction to perform high-quality processing (YES in ST102), the control unit 21 proceeds to the next step, ST103. On the other hand, if the user has not input an instruction to perform high-quality processing (NO in ST102), that is, if low-quality processing is to be performed, the control unit 21 skips steps ST103 to ST125 and proceeds to ST126. In ST126, the control unit 21 performs processing using the Poisson mesh 3D reconstruction method using the current 3D point cloud PG (i.e., a 3D point cloud PG that has not been subjected to interpolation point generation processing, normal vector correction processing, etc.).

[0048] If the user inputs an instruction to execute high-quality processing (YES in ST102), the control unit 21 executes region division processing in the next steps ST103 to ST105. Fig. 8 is a diagram showing the state when the region division processing is executed.

[0049] In the region division process, first, the control unit 21 sets the height coordinate (z value) of each point P in the three-dimensional point group PG to 0, and projects each point P in the three-dimensional point group PG onto a two-dimensional region (xy plane) (ST103).

[0050] The top diagram in Fig. 8 shows the original 3D point cloud PG, and the second diagram from the top shows the 3D point cloud PG projected onto a 2D region (xy plane). Note that for ease of understanding, the top and second diagrams in Fig. 8 have pasted on them the images that were used to generate the 3D point cloud PG, but in reality, the 3D point cloud PG generated from these images is projected onto a 2D region (xy plane).

[0051] After projecting each point P in the three-dimensional point group PG onto a two-dimensional region (z = 0), the control unit 21 divides the two-dimensional region into multiple regions based on the coordinates (x, y) of each point P in the two-dimensional region (xy plane) using quadtree processing, and generates two-dimensional divided regions (ST104).

[0052] The second diagram from the bottom in Fig. 8 shows two-dimensional divided regions generated by quadtree processing. Typically, each of the two-dimensional divided regions has the same shape (e.g., square) and the same area (xy plane) as the other two-dimensional divided regions.

[0053] In addition, for multiple two-dimensional divided areas, the shape of at least some of the two-dimensional divided areas may be different from the shape of the other two-dimensional divided areas, and the area of ​​at least some of the two-dimensional divided areas may be different from the area of ​​the other two-dimensional divided areas.

[0054] Next, for each point P whose height coordinate (z-axis direction) was set to 0, the control unit 21 returns the height coordinate to its original value, expands the two-dimensional divided region into three dimensions, and generates a three-dimensional divided region R. The bottom diagram in Fig. 8 shows the state when the two-dimensional divided region is expanded into three dimensions and the three-dimensional divided region R is generated.

[0055] The three-dimensional divided region R is a rectangular region, and its area in the planar direction (xy plane) is equal to that of the corresponding two-dimensional divided region, and its height (z-axis direction) is the same as the height Zmax of the highest point P among the points P included in the three-dimensional divided region R (or Zmax + α (α: margin)).

[0056] The three-dimensional divided regions R are typically regions obtained by dividing a three-dimensional region in which a three-dimensional point group PG exists by a plane perpendicular to the height direction (z-axis direction). Note that, for multiple three-dimensional divided regions R, the shape of at least some of the three-dimensional divided regions R in a planar direction (xy plane) may be different from the shape of the other three-dimensional divided regions R in the planar direction, and the area of ​​at least some of the three-dimensional divided regions R in the planar direction may be different from the area of ​​the other three-dimensional divided regions R in the planar direction.

[0057] In this embodiment, as shown in ST103 to ST105 and Fig. 8, in the region division process, the three-dimensional point group PG is projected onto a two-dimensional region, the two-dimensional region is divided, and the two-dimensional divided region is then expanded into three dimensions to generate the three-dimensional divided region R. In this case, the three-dimensional divided region R can be generated more quickly than when the three-dimensional region is directly divided to generate the three-dimensional divided region R. Therefore, from the perspective of improving processing speed, it is advantageous to employ the method shown in ST103 to ST105 and Fig. 8 in the region division process.

[0058] However, this does not mean that the present embodiment cannot employ a method of directly dividing a three-dimensional region to generate three-dimensional divided regions R. In other words, in the present embodiment, the control unit 21 may directly divide a three-dimensional region in which a three-dimensional point group PG exists to generate three-dimensional divided regions R. In this case, for example, the control unit 21 divides the three-dimensional region into multiple regions by octree processing based on the coordinates (x, y, z) of each point P in the three-dimensional region, and generates three-dimensional divided regions R.

[0059] After executing the region division process, the control unit 21 executes a three-dimensional divided region selection process in the next steps ST106 to ST113, in which it determines whether the corresponding three-dimensional divided region R is a three-dimensional divided region R that is to be subjected to the interpolation point generation process and the normal vector correction process. In this embodiment, the processes in steps ST106 to ST124 are executed simultaneously in parallel for all three-dimensional divided regions R. This allows the processing to be accelerated. It is also possible to execute the processes for each three-dimensional divided region R in order.

[0060] In the three-dimensional divided region selection process, first, the control unit 21 determines whether the height (z-axis direction) of the corresponding three-dimensional divided region R is equal to or greater than a predetermined threshold value (ST106).

[0061] If the height (z-axis direction) of the three-dimensional divided region R is equal to or greater than the predetermined threshold (YES in ST106), the control unit 21 proceeds to the next step ST107. On the other hand, if the height (z-axis direction) of the three-dimensional divided region R is less than the predetermined threshold (NO in ST106), the control unit 21 determines that the three-dimensional divided region R is not a target for interpolation point generation processing, normal vector correction processing, etc. (ST113).

[0062] 9 is a diagram showing a three-dimensional divided region R whose height is equal to or greater than a predetermined threshold, and a three-dimensional divided region R whose height is less than the predetermined threshold. For example, the three-dimensional divided region R shown on the left side of FIG. 9 has a height equal to or greater than the predetermined threshold, so the processing from ST107 onward is performed on this three-dimensional divided region R. On the other hand, the three-dimensional divided region R shown on the right side of FIG. 9 has a height less than the predetermined threshold, so this three-dimensional divided region R is determined to be a three-dimensional divided region R that is not subject to the interpolation point generation processing, normal vector correction processing, etc.

[0063] The predetermined threshold value in ST106 is set to a height corresponding to a height of about 2 m in the real world. In this case, for example, an object that is 2 m or taller, such as a building (see FIG. 2), can be subject to the interpolation point generation process, normal vector correction process, etc. On the other hand, an object that is less than 2 m, such as a car (see FIG. 2), is not subject to the interpolation point generation process, normal vector correction process, etc.

[0064] If the height (z-axis direction) of the three-dimensional divided region R is equal to or greater than a predetermined threshold (YES in ST106), the control unit 21 divides the three-dimensional divided region R into an upper first region R1 and a lower second region R2 (ST107). The left side of Figure 9 shows the state when the three-dimensional divided region R is divided into the upper first region R1 and the lower second region R2.

[0065] In dividing the three-dimensional divided region R into upper and lower parts, the control unit 21 first calculates the average value Zave (Zave = (Zmax + Zmin) / 2) of the height Zmax of the point P with the highest height (in the z-axis direction) and the height Zmin of the point P with the lowest height among the points P included in the corresponding three-dimensional divided region R. Then, the control unit 21 divides the three-dimensional divided region R into upper and lower parts on a plane that passes through this average value Zave and is parallel to the planar direction (xy plane).

[0066] The method for dividing the three-dimensional divided region R into upper and lower parts is not limited to this. For example, the three-dimensional divided region R may be divided at a position that is 50% of the height Zmax of the point P that has the highest height (in the z-axis direction) among the points P included in the corresponding three-dimensional divided region R.

[0067] The first upper region R1 is a region in which the height is in a first range (e.g., Zave≦z≦Zmax), and the second lower region R2 is a region in which the height is in a second range (e.g., 0≦z<Zave) that is lower than the first range.

[0068] In the following description, in the point cloud PG within the three-dimensional divided region R, points P included in the first upper region R1, i.e., points P whose heights are within a first range, will be referred to as first points P1. On the other hand, in the point cloud PG within the three-dimensional divided region R, points P included in the second lower region R2, i.e., points P whose heights are within a second range, will be referred to as second points P2. Furthermore, the point cloud PG including multiple first points P1 in the upper first region R1 will be referred to as first point cloud PG1, and the point cloud PG including multiple second points P2 in the lower second region R2 will be referred to as second point cloud PG2.

[0069] After dividing the three-dimensional divided region R into upper and lower regions, the control unit 21 next determines whether the number of first points P1 in the upper first region R1 is equal to or greater than a first threshold (ST108). If the number of first points P1 in the first region R1 is equal to or greater than the first threshold (YES in ST108), the control unit 21 proceeds to the next step, ST109. On the other hand, if the number of first points P1 in the first region R1 is less than the first threshold (NO in ST108), the control unit 21 determines that the three-dimensional divided region R is not subject to interpolation point generation processing, normal vector correction processing, etc. (ST113).

[0070] 10 is a diagram illustrating an example of a case where the number of points P included in a three-dimensional divided region R is small. As illustrated in Fig. 10, when the number of first points P1 included in an upper first region R1 is less than a first threshold, the three-dimensional divided region R is determined to be a three-dimensional divided region R that is not subject to interpolation point generation processing, normal vector correction processing, etc., even if the height is equal to or greater than a certain level.

[0071] For example, in the case of a three-dimensional divided region R including the outline of the roof of a building, the number of first points P1 included in the upper first region R1 will be relatively large (because many feature points are extracted from the outline of the roof). Therefore, in the case of such a three-dimensional divided region R, the number of first points P1 in the first region R1 is deemed to be equal to or greater than a first threshold, and the region R1 may be subjected to interpolation point generation processing, normal vector correction processing, and the like.

[0072] On the other hand, for example, in the case of a three-dimensional divided region R corresponding to the vicinity of the center of the roof of a building, the number of first points P1 included in the upper first region R1 will be relatively small (because the number of feature points is smaller than that of the outline of the roof). Therefore, in the case of such a three-dimensional divided region R, the number of first points P1 in the first region R1 is considered to be less than the first threshold, and the region will not be subjected to the interpolation point generation process, normal vector correction process, etc.

[0073] If there is a distinctive pattern or the like near the center of the roof, it is conceivable that the number of first points P1 included in the upper first region R1 in the three-dimensional divided region R corresponding to the center of the roof will be equal to or greater than the first threshold. In this case, the three-dimensional divided region R corresponding to the center of the roof will be excluded from the interpolation point generation process, etc., in ST111 described below (because there are almost no distinctive points in the lower second region R2).

[0074] If the number of first points P1 in the upper first region R1 is equal to or greater than the first threshold (YES in ST108), the control unit 21 generates a surrounding region CR that surrounds the first points P1 in the first region R1 (ST109). Figure 11 is a diagram showing the three-dimensionally divided region R and the surrounding region CR. As shown on the right side of Figure 11, the surrounding region CR is a region in a planar direction (xy plane). For example, a convex hull or the like is used to generate this surrounding region CR, but the surrounding region CR is set to be slightly larger than the region generated by the convex hull.

[0075] After generating the encircled region CR, the control unit 21 then excludes, from among the multiple second points P2 in the lower second region R2, the second point P2 that is located at a position corresponding to the encircled region CR in the planar direction (xy plane) from the target of the processes in ST111 and ST114 described below (ST110). Note that this does not mean that the second point P2 is completely removed from the point cloud PG, but simply that the second point P2 is excluded from the target of the processes in ST111 and ST114 described below (because this would adversely affect the processes). Therefore, the second point P2 excluded from the target of the processes in ST110 may be used, for example, in ST126 described below.

[0076] Next, the control unit 21 determines whether the number of second points P2 in the lower second region R2 is equal to or greater than a second threshold (ST111). Note that the second points P2 that were excluded from processing in ST110 are excluded from this determination. The second threshold may be different from or the same as the first threshold.

[0077] If the number of second points P2 in the second region R2 is equal to or greater than the second threshold (YES in ST111), the control unit 21 determines that the three-dimensional divided region R is a three-dimensional divided region R that is subject to interpolation point generation processing, normal vector correction processing, etc. (ST112). On the other hand, if the number of second points P2 in the second region R2 is less than the second threshold (NO in ST111), the control unit 21 determines that the three-dimensional divided region R is a three-dimensional divided region R that is not subject to interpolation point generation processing, normal vector correction processing, etc. (ST113).

[0078] 9 and 11, in the case of such a three-dimensional divided region R, the number of second points P2 included in the lower second region R2 is deemed to be equal to or greater than the second threshold, and the region may be subject to interpolation point generation processing, normal vector correction processing, etc. On the other hand, in the case of such a three-dimensional divided region R, the number of second points P2 included in the lower second region R2 is deemed to be less than the second threshold, and the region is not subject to interpolation point generation processing, normal vector correction processing, etc.

[0079] For example, in the case of a three-dimensional divided region R that includes the outline of the roof of a building and also includes the ground, the number of first points P1 included in the upper first region R1 is relatively large, and the number of second points P2 included in the lower second region R2 is relatively large (because feature points are extracted on the ground). Therefore, such a three-dimensional divided region R can be subject to interpolation point generation processing, normal vector correction processing, etc.

[0080] On the other hand, for example, in the case of a three-dimensional divided region R corresponding to the vicinity of the center of the roof of a building, the number of first points P1 included in the upper first region R1 is relatively small (although there may be a case where it is large), and the number of second points P2 included in the lower second region R2 is relatively small (because the target of feature point extraction does not exist in the two-dimensional image). Therefore, such a three-dimensional divided region R is not subject to interpolation point generation processing, normal vector correction processing, etc.

[0081] If it is determined in ST113 that the corresponding three-dimensional divided region R is not a target for interpolation point generation processing, normal vector correction processing, etc., the control unit 21 proceeds to ST125 without executing the processing of ST114 to ST124 described below. On the other hand, if it is determined that the corresponding three-dimensional divided region R is a target for interpolation point generation processing, normal vector correction processing, etc., the control unit 21 proceeds to the next ST114.

[0082] Referring to Figure 7, after the selection process of the three-dimensional divided region R, the control unit 21 performs a normal vector correction process in the next ST114 to ST121 to correct the normal vector at a predetermined first point P1 among the multiple first points P1 in the upper first region R1.

[0083] FIG. 12 is a diagram for explaining the normal vector correction process and the interpolation point generation process.

[0084] In the normal vector correction process, first, in ST114, the control unit 21 performs pairing between a first point P1 in the first upper region R1 and a second point P2 in the second lower region R2. The third diagram from the left in Fig. 12 shows how this pairing is performed.

[0085] In the pairing, first, the control unit 21 sets the height coordinates (z-axis direction) of all first points P1 in the upper first region R1 and all second points P2 in the lower second region R2 to 0, and projects these onto a two-dimensional region (xy plane). Then, the control unit 21 determines, for each of all first points P1 in the two-dimensional region (xy plane), the second point P2 that is located closest to the first point P1 using a kd tree (k-dimensional tree) or the like, and pairs the first point P1 with the second point P2 that is closest to the first point P1.

[0086] Next, the control unit 21 calculates the paired first points P1(x １ , y １ ) to the second point P2 (x ２ , y ２ ) pair vector (xn, yn) = (x ２ -x １ , y ２ -y １ ) is calculated (ST115). Note that this calculation is performed in a two-dimensional area (xy plane), so the pair vectors at this point P are two-dimensional.

[0087] In this pairing, the second point P2 determined to be located within the above-described encircling region CR is excluded from the second points P2 to be paired, thereby appropriately preventing the orientation of the pair vectors that are the basis for correcting the normal vector from being oriented in the wrong direction.

[0088] Next, the control unit 21 calculates a centroid vector. The second diagram from the left in FIG. 12 shows how this centroid vector is calculated. In calculating the centroid vector, the control unit 21 first calculates the first centroid position C1(xc １ , yc １ ), and the second center of gravity position C2(xc ２ , yc ２ ) are calculated (ST116).

[0089] Then, the control unit 21 calculates the first center of gravity position C1(xc １ , yc １ ) to the second center of gravity position C2 (xc ２ , yc ２ ) and the center of gravity vector (xc, yc) = (xc ２ -xc １ , yc ２ -yc １ ) is calculated (ST117). Note that this calculation is performed within a two-dimensional area (within the xy plane), so the centroid vector is two-dimensional.

[0090] Next, the control unit 21 calculates the angle between each pair of vectors and the center of gravity vector for each pair of vectors.The control unit 21 then reverses the direction of each pair of vectors by 180 degrees for those vectors whose direction is at a predetermined angle or more with respect to the direction of the center of gravity vector (ST118).The predetermined angle is, for example, about 90 degrees, but this angle can be changed as appropriate.

[0091] The third diagram from the left in Figure 12 shows what happens when the vector direction of a pair of vectors, whose vector direction is greater than a predetermined angle (e.g., 90°) with respect to the direction of the center of gravity vector, is reversed by 180°.

[0092] As described above, since the second point P2 located within the enclosing region CR is excluded from the second points P2 to be paired, it is possible to prevent the pair vectors that are the basis for correcting the normal vector from pointing in an inappropriate direction. However, as shown in the third diagram from the left in Figure 12, even if the second point P2 located within the enclosing region CR is excluded from the second points P2 to be paired, the pair vectors may still point in an inappropriate direction (for example, the opposite direction to the normal to the wall of the building).

[0093] Therefore, in this embodiment, the direction of the pair vectors that are facing in an inappropriate direction is corrected using the center of gravity vector, which makes it possible to more appropriately prevent the direction of the pair vectors that are the basis for correcting the normal vector from facing in an inappropriate direction.

[0094] After correcting the orientation of the pair vectors, the control unit 21 normalizes each of all pair vectors and adds z=0 to the height coordinate of the vector to make the pair vectors three-dimensional (xn, yn, 0) (ST119). At this time, the control unit 21 also returns the height coordinates (z-axis direction) of all first points P1 and all second points P2 to their original values, restoring each point P to three dimensions.

[0095] Next, the control unit 21 sorts the pairs in descending order of the distance (two-dimensional distance: see ST114) between the first point P1 and the second point P2, and determines the top n% of pairs with the closest distance (ST120). Next, the control unit 21 corrects the normal vector of the first point P1 by replacing the current normal vector of the first point P1 in the top n% of pairs with the pair vector (xn, yn, 0) corresponding to that pair (ST121). Note that the corrected normal vector (xn, yn, 0) has a z value of 0, and is a vector parallel to the planar direction (xy plane).

[0096] The rightmost diagram in Figure 12 shows the state when the normal vectors of the top n% of first points P1 among all first points P1 included in the first point group PG1 in the upper first region R1 are replaced and corrected by the pair vector (xn, yn, 0).

[0097] In this example, the number of first points P1 whose normal vectors are corrected (the number of normal correction points RP) is variable for each three-dimensional divided region R (because they are the top n%). On the other hand, the number of first points P1 whose normal vectors are corrected may be constant for each three-dimensional divided region R. In this case, the number of first points P1 whose normal vectors are corrected is set in advance (one, two, ..., for each three-dimensional divided region R).

[0098] In this example, the normal vector of the first point P1 is corrected by replacing it with the pair vector (xn, yn, 0). Alternatively, the normal vector of the first point P1 may be corrected by replacing it with the center of gravity vector (xc, yc, 0) (in this case, z=0 is added).

[0099] 13 is a perspective view of a building, showing normal correction points RP whose normal vectors have been corrected and interpolation points IP. In FIG. 13, the area where the normal correction points RP whose normal vectors have been corrected are concentrated is surrounded by a rectangle.

[0100] Fig. 14 is a diagram showing a normal map corresponding to Fig. 13. In Fig. 14, white areas indicate normals perpendicular to the ground (z-axis direction), and black areas indicate normals parallel to the ground (xy plane direction). Fig. 15 is a top view of a building, showing normal correction points RP. In Fig. 15, the area where normal correction points RP with corrected normal vectors are concentrated is surrounded by a rectangle.

[0101] 13 and 15, the area where normal correction points RP (part of the first point P1) whose normal vectors have been corrected are concentrated is surrounded by a rectangle. As shown in FIGS. 13 and 15, it can be seen that the normal correction points RP are concentrated in the area corresponding to the corner where the roof of the building and the wall surface join. Furthermore, in the normal map shown in FIG. 14, the area corresponding to the corner where the roof of the building and the wall surface join is black, which indicates that the normal vector at the area corresponding to this corner is oriented in a direction parallel to the plane direction (xy plane direction) rather than the height direction (z-axis direction) (black: normal vector is in the plane direction (xy plane); white: normal vector is in the height direction (z-axis direction)).

[0102] That is, in this embodiment, each point P (part of the first point P1) corresponding to the corner (roof outline) where the roof of the building and the wall surface are connected can be appropriately detected as a normal correction point RP. Also, in this embodiment, the normal vector of the normal correction point RP can be directed in a direction parallel to the plane direction and perpendicular to the wall surface (as described in FIG. 4, the normal vector of point P corresponding to the corner tends to point diagonally upward before correction).

[0103] 16 is a top view showing the terrain, illustrating normal correction points RP. It can be seen from FIG. 16 that, for each building, points P corresponding to the corners where the roof of the building connects to the wall (roof outline) are appropriately detected as normal correction points RP, and the normal direction is appropriately corrected. Note that in this embodiment, points P corresponding to the tops of objects other than buildings, such as trees, may also be detected as normal correction points RP, and the normal vector may be corrected.

[0104] When the normal vector correction process is completed, the control unit 21 executes the interpolation point generation process in the next steps ST122 to ST124. In the interpolation point generation process, the control unit 21 first sorts pairs of points P1 and P2 in descending order of distance (two-dimensional distance: see ST114) between them, and determines the top m% of pairs with the shortest distance (ST122). Note that the value of m is typically set to a value equal to or less than the value of n (m≦n), but may also be a value greater than n (m>n).

[0105] Next, the control unit 21 arranges interpolation points IP at predetermined intervals downward in the height direction (z-axis direction) from the first point P1 in the top m% of pairs with the closest distance (ST123). As a result, multiple interpolation points IP corresponding to, for example, the wall surfaces of a building are newly arranged.

[0106] The rightmost diagram in Figure 12 shows the state when interpolation points IP are arranged at a predetermined interval from the top m% of first points P1 of all first points P1 included in the first point group PG1 in the upper first region R1 downward.

[0107] Here, the interval between the interpolation points IP adjacent to each other in the height direction may be constant in the height direction (z-axis direction) or may be variable in the height direction. In other words, the control unit 21 may arrange the interpolation points IP with a constant interval between the interpolation points IP in the height direction, or may arrange the interpolation points IP with a variable interval between the interpolation points IP in the height direction.

[0108] When the intervals between the interpolation points IP are changed, for example, the intervals between the interpolation points IP located at a position equal to or higher than a predetermined height may be narrower than the intervals between the interpolation points IP located at a position below the predetermined height. In this case, for example, in the model (mesh), the corners where the roof and wall surfaces of a building join can be made to stand out as right angles.

[0109] In this example, the number of columns of interpolation points IP is variable for each three-dimensional divided region R (because they are in the top m%). On the other hand, the number of columns of interpolation points IP may be constant for each three-dimensional divided region R. In this case, the number of columns of interpolation points IP is set in advance (one column, two columns, etc. for each three-dimensional divided region R).

[0110] After placing the interpolation point IP, the control unit 21 then sets the normal vector of the interpolation point IP as the same vector (pair vector (xn, yn, 0)) as the corrected normal vector at the corresponding first point P1 (the first point P1 directly above the interpolation point IP) (ST124). This sets the normal vector for the newly placed interpolation point IP. Note that the normal vector (pair vector (xn, yn, 0)) of the interpolation point IP also has a z value of 0, and is a vector parallel to the planar direction (xy plane).

[0111] In this example, the normal vector set to the interpolation point IP is a pair vector (xn, yn, 0). On the other hand, the normal vector set to the interpolation point IP may be a center of gravity vector (xc, yc, 0) (in this case, z=0 is added).

[0112] 13, interpolation points IP are arranged at predetermined intervals along the height direction at positions corresponding to the wall surfaces of the building. In this manner, in this embodiment, the interpolation points IP can be appropriately arranged at positions corresponding to the wall surfaces of the building. Furthermore, in the normal map shown in FIG. 14, the areas corresponding to the interpolation points IP are black, which indicates that the normal vectors of the areas corresponding to the wall surfaces are appropriately oriented perpendicular to the wall surfaces (black: normal vector is in the planar direction (xy plane); white: normal vector is in the height direction (z-axis direction)).

[0113] After executing the interpolation point generation process, the control unit 21 then determines whether the three-dimensional divided area selection process, normal vector correction process (for some three-dimensional divided areas R), and interpolation point generation process (for some three-dimensional divided areas R) have been completed for all three-dimensional divided areas R (ST125).

[0114] When each process has been completed for all three-dimensional divided regions R (YES in ST125), the control unit 21 uses the three-dimensional point group PG after normal vector correction and addition of interpolation points IP to three-dimensionally reconstruct the model using the Poisson mesh reconstruction method (meshing) (ST126).

[0115] 17 is a diagram showing a state in which a building is three-dimensionally reconstructed by the Poisson mesh reconstruction method (meshing). As shown in FIG. 17, according to this embodiment, the wall surfaces of the building can be accurately reproduced.

[0116] After three-dimensionally reconstructing the model, the control unit 21 may perform orthoimage processing (orthogonal projection transformation) on the aerial photographed image based on the three-dimensional model representing the earth's surface to generate an orthoimage.

[0117] Fig. 18 is a diagram showing an orthoimage. In this embodiment, the wall surfaces of a building can be accurately reproduced in the 3D reconstruction (meshing) of the model, so that an accurate orthoimage can be generated without distortion occurring in the portions of the orthoimage corresponding to the wall surfaces, as shown in Fig. 18.

[0118] <Functions, etc.> Next, the functions, effects, etc. of this embodiment will be described. Fig. 19 is a diagram for explaining the functions, etc. of this embodiment.

[0119] The first example in Figure 19 shows an example of a case where a three-dimensional point group PG (top row) is used in a case where neither the interpolation point generation process nor the normal vector correction process is performed, and meshing (middle row) is performed using Poisson mesh reconstruction, and an orthoimage (bottom row) is generated.

[0120] The second example in Figure 19 shows an example of a case where meshing (middle row) is performed using Poisson mesh reconstruction using a three-dimensional point group PG (top row) when only the interpolation point generation process is executed out of the interpolation point generation process and the normal vector correction process, and an orthoimage (bottom row) is generated.

[0121] The third example in Figure 19 shows an example of a case where meshing (middle row) is performed using Poisson mesh reconstruction using a three-dimensional point group PG (top row) when only the normal vector correction process is performed out of the interpolation point generation process and the normal vector correction process, and an orthoimage (bottom row) is generated.

[0122] 19 shows an example of a case where meshing (middle part) is performed by Poisson mesh reconstruction using a 3D point cloud PG (top part) in which both the interpolation point generation process and the normal vector correction process have been performed, and an orthoimage (bottom part) is generated. Note that this fourth example corresponds to the form described above using FIGS. 6 and 7, etc.

[0123] In the first example, the 3D point cloud PG is almost nonexistent in the area corresponding to the building wall (top), so the mesh in the area corresponding to the wall is significantly distorted (middle). As a result, the area corresponding to the wall bulges outward in the orthoimage (bottom).

[0124] In the second example, the interpolation point IP is located at a location corresponding to the wall surface of the building, and a normal vector perpendicular to the wall surface is set for the interpolation point IP (top row). Therefore, the distortion of the mesh at the location corresponding to the wall surface is smaller than in the first example (middle row). Therefore, the bulge at the location corresponding to the wall surface in the orthoimage is smaller than in the first example (bottom row).

[0125] In the third example, the normal vector of the point cloud PG at the corner where the roof and wall of the building meet is corrected to point perpendicular to the wall (top), so the distortion of the mesh at the wall is smaller than in the first and second examples (middle). As a result, the bulge at the wall in the orthoimage is smaller than in the first and second examples (bottom).

[0126] In the fourth example, an interpolation point IP is placed at a location corresponding to the building's wall, and a normal vector perpendicular to the wall is set relative to the interpolation point IP (top). Also, in the fourth example, the normal vector of the point cloud PG at the corner where the building's roof and wall connect is corrected to point perpendicular to the wall (top). Therefore, the mesh distortion at the location corresponding to the wall is smaller than in the first to third examples (middle). Therefore, in the orthoimage, the bulge at the location corresponding to the wall is smaller than in the first to third examples, and almost no bulge occurs on the wall (bottom).

[0127] In this embodiment, the case where both the interpolation point generation process and the normal vector correction process are executed has been described, as in the fourth example. On the other hand, it is also possible to execute only the interpolation point generation process out of the interpolation point generation process and the normal vector correction process, as in the second example. Even in this case, the model can be reconstructed in three dimensions more accurately, and an orthoimage can be generated more accurately than when the three-dimensional point group PG is used as is (first example).

[0128] Furthermore, as in the third example, it is also possible to execute only the normal vector correction process out of the interpolation point generation process and the normal vector correction process. In this case, too, the model can be reconstructed in three dimensions more accurately, and an orthoimage can be generated more accurately than when the three-dimensional point cloud PG is used as is (first example).

[0129] Fig. 20 is another diagram for explaining the operation of this embodiment. The image on the left in Fig. 20 is an image according to a comparative example. This image on the left is an image generated by Delaunay mesh reconstruction based on a point cloud PG (original point cloud PG) that has not been subjected to interpolation point generation processing, normal vector correction processing, etc.

[0130] In this Delaunay mesh reconstruction, the meshing process took 263 seconds, and the texturing process (the process of attaching a partial image cut out from an aerial image to the mesh) took 190 seconds.

[0131] 20 is an image according to this embodiment, which is generated by Poisson mesh reconstruction based on the point group PG that has been subjected to the interpolation point generation process and normal vector correction process according to this embodiment.

[0132] In this Poisson mesh reconstruction, the meshing process took 179 seconds, and the texturing process (the process of attaching a partial image cut out from an aerial image to the mesh) took 111 seconds.

[0133] A comparison of the image created by Delaunay mesh reconstruction based on the original point cloud PG shown on the left side of Fig. 20 with the image created by Poisson mesh reconstruction based on the point cloud PG processed according to this embodiment shown on the right side of Fig. 20 reveals that the images are of similar accuracy. This means that this embodiment can create an image of similar accuracy to Delaunay mesh reconstruction in a shorter time than Delaunay mesh reconstruction.

[0134] In this embodiment, processing such as interpolation point generation processing is performed on the 3D point group PG, whereas in the comparative example, processing such as interpolation point generation processing is not performed on the 3D point group PG. Therefore, in order to compare the time required to create the image (left) in the comparative example with the time required to create the image (right) in this embodiment, it is necessary to take into account the time required for this processing. However, even when this time is taken into account, the image can be generated in a shorter time in this embodiment than in the comparative example.

[0135] As described above, in this embodiment, a plurality of interpolation points IP are generated for interpolating the plurality of points P in the height direction based on a plurality of first points P1 whose height coordinates are within a first range and a plurality of second points P2 whose height coordinates are within a second range lower than the second range, among the plurality of points P in the three-dimensional point cloud PG. This makes it possible to appropriately generate interpolation points IP for locations that do not appear in the original two-dimensional image (aerial image), such as those corresponding to the walls of a building. This allows for accurate three-dimensional reconstruction of the model.

[0136] Furthermore, in this embodiment, a plurality of interpolation points IP are generated for each three-dimensional divided region R based on a plurality of first points P1 and a plurality of second points P2 for each three-dimensional divided region R. This allows the interpolation points IP to be generated appropriately.

[0137] Furthermore, in this embodiment, in generating the three-dimensional divided region R, first, as a preliminary step to generating the three-dimensional divided region R, a two-dimensional region in the planar direction is divided into multiple parts to generate multiple two-dimensional divided regions. Then, the two-dimensional divided regions are expanded (restored) into three dimensions to generate the three-dimensional divided region R. By generating the three-dimensional divided region R in this manner, the processing speed can be improved compared to when the three-dimensional divided region R is directly generated in three dimensions.

[0138] Furthermore, in this embodiment, it is determined whether the corresponding three-dimensional divided region R is a three-dimensional divided region R for which interpolation points IP should be generated and normal vectors should be corrected, based on the height size of the three-dimensional divided region R. This makes it possible to appropriately select a three-dimensional divided region R for which interpolation points IP should be generated and normal vectors corrected (for example, a three-dimensional divided region corresponding to a building).

[0139] Furthermore, in this embodiment, based on the number of points P included in the three-dimensional divided region R, it is determined whether the corresponding three-dimensional divided region R is a three-dimensional divided region R for which interpolation points IP should be generated. In particular, in this embodiment, if the number of first points P1 included in the three-dimensional divided region R is equal to or greater than a predetermined threshold and the number of second points P2 included in the three-dimensional divided region R is equal to or greater than a predetermined threshold, the corresponding three-dimensional divided region R is determined to be a three-dimensional divided region R for which interpolation points IP should be generated and normal vectors should be modified. This makes it possible to appropriately select three-dimensional divided regions R for which interpolation points IP should be generated and normal vectors should be modified (e.g., three-dimensional divided regions including the wall surfaces of a building).

[0140] Furthermore, in this embodiment, the normal vector of a predetermined point P among the plurality of second points P2 is corrected based on the plurality of first points P1 and the plurality of second points P2. This makes it possible to correct the normal vector at a location where the normal vector is likely to point in the wrong direction, for example, at a location corresponding to a corner where the roof and wall surfaces of a building are connected.

[0141] In this embodiment, for each of the multiple first points P1, a second point P2 that is paired with the first point P1 is determined, and the normal vector of the corresponding first point P1 is corrected based on the paired first point P1 and second point P2. This makes it possible to appropriately correct the direction of the normal vector at the first point P1.

[0142] In this embodiment, the control unit 21 determines a surrounding area CR in a planar direction that surrounds the multiple first points P1, and excludes, from the multiple second points P2, the second points P2 that are located at positions that overlap with the surrounding area CR in the planar direction, from the second points P2 to be paired. This prevents the pairing vector, which is the basis for correcting the normal vector, from being oriented in an inappropriate direction.

[0143] In this embodiment, a vector that is a direction from the paired first point P1 to the paired second point P2 and is parallel to the plane direction is set as a pair vector, and the normal vector is corrected based on this pair vector, thereby making it possible to correct the normal vector to an appropriate direction.

[0144] In this embodiment, the first center-of-gravity positions C1 of the first points P1 and the second center-of-gravity positions C2 of the second points P2 are determined, and a vector extending from the first center-of-gravity positions C1 to the second center-of-gravity positions C2 and parallel to the plane direction is determined as a center-of-gravity vector, and the orientation of the pair vectors is corrected based on this center-of-gravity vector. This prevents the pairing vector, which is the basis for correcting the normal vector, from being oriented in an inappropriate direction.

[0145] In this embodiment, if the orientation of the center of gravity vector is greater than or equal to a predetermined angle (e.g., 90°) with respect to the orientation of the pair vector, the orientation of the pair vector is reversed, thereby allowing the pairing vector, which is the source of the normal vector correction, to be oriented appropriately.

[0146] In this embodiment, interpolation points IP are arranged at predetermined intervals in the downward vertical direction from a predetermined first point P1 among the plurality of first points P1. This makes it possible to appropriately generate interpolation points IP for locations that do not appear in the original two-dimensional image (aerial image), such as those corresponding to the wall surfaces of a building.

[0147] Furthermore, in this embodiment, the spacing between interpolation points IP in the height direction may not be constant, but may vary in the height direction. In this case, in the model (mesh), for example, corners where the roof and wall surfaces of a building are connected can be made to stand out as right angles.

[0148] In this embodiment, the normal vector of the interpolation point IP is set to be the same vector as the corrected normal vector of the corresponding first point P1, thereby allowing the normal vector of the interpolation point IP to be oriented in an appropriate direction.

[0149] <<Various Modifications>> In the above explanation, a case where Poisson mesh reconstruction is used in 3D reconstruction has been described, but other methods such as Delaunay mesh reconstruction may also be used (in the case of a Delaunay mesh, normal vectors are not necessarily required, so in this case, for example, only the interpolation point generation process is performed).

[0150] The present technology may also be configured as follows. (1) An information processing device including a control unit that generates, among a plurality of points used for 3D reconstruction of a model, a plurality of first points whose height coordinates are within a first range and a plurality of second points whose height coordinates are within a second range lower than the first range, a plurality of interpolation points for interpolating the plurality of points in a height direction, each of the first points including information on a three-dimensional coordinate position. (2) The information processing device described in (1), in which the control unit generates a plurality of 3D divided regions by dividing the three-dimensional region by planes parallel to the height direction, and generates the plurality of interpolation points for each of the 3D divided regions based on the plurality of first points and the plurality of second points for each of the 3D divided regions. (3) The information processing device described in (2), in which the control unit divides a two-dimensional region in a planar direction into a plurality of parts and generates a plurality of 2D divided regions as a preliminary step to generating the three-dimensional divided regions. (4) The information processing device according to (3) above, wherein the control unit sets height coordinates of the plurality of points to 0, projects the plurality of points onto the two-dimensional area, and divides the two-dimensional area into a plurality of areas based on the coordinates of the plurality of points within the two-dimensional area. (5) The information processing device according to (4) above, wherein the control unit expands the two-dimensional divided area into three dimensions to generate a three-dimensional divided area. (6) The information processing device according to (4) or (5), wherein the control unit restores height coordinates of the plurality of points whose height coordinates were set to 0 to their original values, and then generates the plurality of interpolation points based on the plurality of first points and the plurality of second points. (7) The information processing device according to any one of (2) to (6), wherein the control unit determines, based on a height size of the corresponding three-dimensional divided area, whether the corresponding three-dimensional divided area is a three-dimensional divided area for which the interpolation points should be generated. (8) An information processing device according to any one of (2) to (7) above, wherein the control unit determines, based on the number of points included in the corresponding three-dimensional divided area, whether the corresponding three-dimensional divided area is a three-dimensional divided area that should be the target for generating the interpolation points.(9) The information processing device according to (8) above, wherein the control unit determines that the corresponding three-dimensional divided area is a three-dimensional divided area for which the interpolation points should be generated when the number of a plurality of first points included in the three-dimensional divided area is equal to or greater than a predetermined threshold and the number of the plurality of second points is equal to or greater than a predetermined threshold. (10) The information processing device according to any one of (2) to (9) above, wherein the plurality of points each include information on a normal vector, and the control unit corrects the normal vector of a predetermined point among the plurality of first points based on the plurality of first points and the plurality of second points. (11) The information processing device according to (10) above, wherein the control unit determines, for each first point among the plurality of first points, a second point paired with the first point, and corrects the normal vector of the corresponding first point based on the paired first point and the second point. (12) The information processing device according to (11), wherein the control unit determines an enclosing area in a planar direction that encloses the plurality of first points, and excludes, from the plurality of second points, second points that are located at positions that overlap with the enclosing area in the planar direction from the second points to be paired. (13) The information processing device according to (11) or (12), wherein the control unit sets, as a pair vector, a vector that is in a direction from the paired first point to the second point and that is parallel to the planar direction, and corrects the normal vector based on the pair vector. (14) The information processing device according to (13), wherein the control unit determines first centroid positions of the plurality of first points and second centroid positions of the plurality of second points, and sets, as a centroid vector, a vector that is in a direction from the first centroid position to the second centroid position and that is parallel to the planar direction, and corrects the orientation of the pair vector based on the centroid vector. (15) The information processing device according to (14), wherein the control unit reverses the direction of the pair vector when the direction of the center of gravity vector is equal to or greater than a predetermined angle with respect to the direction of the pair vector.(16) The information processing device according to any one of (10) to (15), wherein the control unit arranges the interpolation points at predetermined intervals downward in the height direction from a predetermined first point among the plurality of first points. (17) The information processing device according to (16), wherein the control unit arranges the interpolation points by changing the predetermined intervals in the height direction. (18) The information processing device according to (16) or (17), wherein the control unit sets a normal vector of the interpolation point to be the same as a corrected normal vector of the corresponding first point. (19) An information processing device comprising: a control unit including information on three-dimensional coordinate positions and information on normal vectors, respectively, and correcting the normal vector of a predetermined point among the plurality of first points based on a plurality of first points whose height coordinates are within a first range and a plurality of second points whose height coordinates are within a second range lower than the first range, among a plurality of points used for three-dimensional reconstruction of a model. (20) An information processing method for generating a plurality of interpolation points for interpolating a plurality of points in a height direction based on a plurality of first points whose height coordinates are within a first range and a plurality of second points whose height coordinates are within a second range lower than the first range, among a plurality of points used for three-dimensional reconstruction of a model, each of which includes information on a three-dimensional coordinate position.

[0151] P...point P1...first point P2...second point PG...three-dimensional point group PG1...first point group PG2...second point group IP...interpolation point RP...normal correction point R...three-dimensional divided region R1...first region R2...second region

Claims

1. An information processing device having a control unit that generates a plurality of interpolation points for interpolating a plurality of points in the height direction based on a plurality of first points whose height coordinates are within a first range and a plurality of second points whose height coordinates are within a second range lower than the first range, each of which contains information on three-dimensional coordinate positions and is used for three-dimensional reconstruction of a model.

2. An information processing device according to claim 1, wherein the control unit generates a plurality of three-dimensional divided areas by dividing a three-dimensional area by planes parallel to the height direction, and generates the plurality of interpolation points for each of the three-dimensional divided areas based on the plurality of first points and the plurality of second points for each of the three-dimensional divided areas.

3. An information processing device according to claim 2, wherein the control unit divides a two-dimensional area in a planar direction into a plurality of areas as a preliminary step to generating the three-dimensional divided areas, thereby generating a plurality of two-dimensional divided areas.

4. An information processing device according to claim 3, wherein the control unit projects the plurality of points onto the two-dimensional area, setting the height coordinates of each of the plurality of points to 0, and divides the two-dimensional area into a plurality of areas based on the coordinates of the plurality of points within the two-dimensional area.

5. An information processing device according to claim 4, wherein the control unit develops the two-dimensional divided area into three dimensions to generate a three-dimensional divided area.

6. An information processing device according to claim 4, wherein the control unit returns the height coordinates of the plurality of points whose height coordinates have been set to 0 to their original values, and then generates the plurality of interpolated points based on the plurality of first points and the plurality of second points.

7. An information processing device according to claim 2, wherein the control unit determines, based on the height size of the corresponding three-dimensional divided area, whether the corresponding three-dimensional divided area is a three-dimensional divided area that should be the target for generating the interpolation points.

8. An information processing device according to claim 2, wherein the control unit determines, based on the number of points included in the three-dimensional divided area, whether the corresponding three-dimensional divided area is a three-dimensional divided area that should be the target for generating the interpolation points.

9. An information processing device according to claim 8, wherein the control unit determines that the corresponding three-dimensional divided area is a three-dimensional divided area that should be the target for generating the interpolation points when the number of multiple first points included in the three-dimensional divided area is equal to or greater than a predetermined threshold and the number of multiple second points is equal to or greater than a predetermined threshold.

10. An information processing device according to claim 2, wherein the plurality of points each include information on a normal vector, and the control unit corrects the normal vector of a predetermined point among the plurality of first points based on the plurality of first points and the plurality of second points.

11. An information processing device according to claim 10, wherein the control unit determines, for each of the plurality of first points, a second point that pairs with the first point, and corrects the normal vector of the corresponding first point based on the paired first point and second point.

12. An information processing device according to claim 11, wherein the control unit determines an enclosed area in a planar direction that surrounds the plurality of first points, and excludes, from the second points to be paired, second points among the plurality of second points that are located at positions that overlap with the enclosed area in the planar direction.

13. An information processing device according to claim 11, wherein the control unit defines a vector in a direction from the paired first point to the paired second point and parallel to the plane direction as a pair vector, and corrects the normal vector based on the pair vector.

14. An information processing device according to claim 13, wherein the control unit determines a first center of gravity position of the plurality of first points and a second center of gravity position of the plurality of second points, defines a vector in a direction from the first center of gravity position to the second center of gravity position and parallel to the plane direction as a center of gravity vector, and corrects the direction of the pair vectors based on the center of gravity vector.

15. An information processing device according to claim 14, wherein the control unit reverses the direction of the pair vector when the direction of the center of gravity vector is at a predetermined angle or more with respect to the direction of the pair vector.

16. An information processing device according to claim 10, wherein the control unit arranges the interpolation points at predetermined intervals downward in the height direction from a predetermined first point among the plurality of first points.

17. An information processing device according to claim 16, wherein the control unit arranges the interpolation points by changing the predetermined interval in the height direction.

18. An information processing device according to claim 16, wherein the control unit sets the normal vector of the interpolation point to be the same vector as the corrected normal vector at the corresponding first point.

19. An information processing device comprising a control unit that includes information on three-dimensional coordinate positions and normal vector information, and that corrects the normal vector of a predetermined point among a plurality of first points used in three-dimensional reconstruction of a model, based on a plurality of first points whose height coordinates are within a first range and a plurality of second points whose height coordinates are within a second range lower than the first range.

20. An information processing method for generating a plurality of interpolation points for interpolating a plurality of points in the height direction based on a plurality of first points whose height coordinates are within a first range and a plurality of second points whose height coordinates are within a second range lower than the first range, each of which contains information on three-dimensional coordinate positions and is used for three-dimensional reconstruction of a model.

Images