Three-dimensional feature change detection method and program

Abstract

To provide a three-dimensional feature change detection method and a program capable of detecting the change of a three-dimensional feature with high accuracy.SOLUTION: The three-dimensional feature change detection method executed by a three-dimensional feature change detection system 1 extracts an upper surface area of a three-dimensional feature based on an image obtained by imaging a region including the three-dimensional feature from above (Step S1). The change detection method calculates a deviation (positional deviation vector) between a footprint on the ground surface of the three-dimensional feature and an upper surface area based on height information of the three-dimensional feature from the ground surface and imaging conditions when a picked-up image is picked up (Step S4). In a state where the deviation between the footprint and the upper surface area is canceled, the change detection method compares the upper surface area of the three-dimensional feature extracted from the picked-up image picked up at a first time point with an area corresponding to the three-dimensional feature at a second time point different from the first time point (Step S8). When the upper surface area is different from the area of the three-dimensional feature, the change detection method determines that there is a change in the three-dimensional feature (Step S9).SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to a method and program for detecting changes in three-dimensional features. [Background technology]

[0002] Technologies for detecting "changes" such as the construction / demolition of three-dimensional features such as buildings or roads between two different time periods are being utilized for various purposes, such as map data maintenance, disaster damage assessment, area marketing, etc. One method for detecting changes in three-dimensional features is disclosed as using aerial photographs and position information of three-dimensional features that have been developed in the past (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2002-297641 Summary of the Invention [Problem to be solved by the invention]

[0004] To detect changes in three-dimensional features, a differential image is generated between a building on a map image and a building captured in an aerial photograph. This is because only buildings that have changed remain in this differential image. However, in an aerial photograph, an image of a tall building appears tilted in a certain direction. This tilt causes the position of the building on the map and the aerial photograph to shift, leaving part of the building in the differential image. This part becomes noise when detecting changes in three-dimensional features.

[0005] The present invention has been made in light of the above-mentioned circumstances, and has as its object to provide a method and program for detecting changes in three-dimensional features that can detect changes in three-dimensional features with high accuracy. [Means for solving the problem]

[0006] In order to achieve the above object, a three-dimensional feature change detection method according to a first aspect of the present invention comprises: A three-dimensional feature change detection method executed by an information processing device, comprising: an extraction step of extracting an upper surface area of ​​the three-dimensional feature based on a captured image obtained by capturing an image of an area including the three-dimensional feature from above; a calculation step of calculating a deviation between the footprint of the three-dimensional feature on the ground surface and the upper surface area based on height information of the three-dimensional feature from the ground and imaging conditions when the captured image was taken; a comparison step of comparing the top surface area of ​​the three-dimensional feature extracted from the captured image taken at a first time point with an area corresponding to the three-dimensional feature at a second time point different from the first time point, while canceling any deviation between the footprint and the top surface area; a determining step of determining that a change has occurred in the three-dimensional feature when the comparison step shows that the upper surface area and the area corresponding to the three-dimensional feature are different; It has.

[0007] A program according to a second aspect of the present invention comprises: On the computer, an extraction step of extracting an upper surface area of ​​the three-dimensional feature based on a captured image obtained by capturing an image of an area including the three-dimensional feature from above; a calculation step of calculating a deviation between the footprint of the three-dimensional feature on the ground surface and the upper surface area based on height information of the three-dimensional feature from the ground and imaging conditions when the captured image was taken; a comparison step of comparing the top surface area of ​​the three-dimensional feature extracted from the captured image taken at a first time point with an area corresponding to the three-dimensional feature at a second time point different from the first time point, while canceling any deviation between the footprint and the top surface area; a determining step of determining that a change has occurred in the three-dimensional feature when the comparison step shows that the upper surface area and the area corresponding to the three-dimensional feature are different; Execute the following. [Effects of the Invention]

[0008] According to the present invention, changes in three-dimensional features can be detected with high accuracy. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram showing the functional configuration of a three-dimensional feature change detection system according to a first embodiment of the present invention. [Figure 2] FIG. 2 is a schematic diagram showing a first data flow in the three-dimensional feature change detection system of FIG. 1. [Figure 3] FIG. 10 is a schematic diagram showing an example of processing for excluding side regions. [Figure 4] FIG. 2 is a schematic diagram showing a second data flow in the three-dimensional feature change detection system of FIG. 1. [Figure 5] FIG. 2 is a block diagram showing the hardware configuration of the three-dimensional feature change detection system of FIG. 1. [Figure 6] 2 is a flowchart of a three-dimensional feature change detection process executed by the three-dimensional feature change detection system of FIG. 1. [Figure 7] FIG. 10 is a block diagram showing the functional configuration of a three-dimensional feature change detection system according to a second embodiment of the present invention. [Figure 8] FIG. 8 is a schematic diagram showing a first data flow in the three-dimensional feature change detection system of FIG. 7. [Figure 9] FIG. 8 is a schematic diagram showing a second data flow in the three-dimensional feature change detection system of FIG. 7. [Figure 10] 8 is a flowchart of a three-dimensional feature change detection process executed by the three-dimensional feature change detection system of FIG. 7. [Figure 11] FIG. 10 is a block diagram showing the functional configuration of a three-dimensional feature change detection system according to a third embodiment of the present invention. [Figure 12] 12 is a schematic diagram showing how the deviation of the upper surface area is cancelled in the three-dimensional feature change detection system of FIG. 11. FIG. [Figure 13] 12 is a flowchart of a three-dimensional feature change detection process executed by the three-dimensional feature change detection system of FIG. 11. [Figure 14] FIG. 10 is a block diagram showing the functional configuration of a three-dimensional feature change detection system according to a fourth embodiment of the present invention. [Figure 15] 15(A) to 15(C) are schematic diagrams showing the data flow in the three-dimensional feature change detection system of FIG. 14. [Figure 16] 15 is a flowchart of a three-dimensional feature change detection process executed by the three-dimensional feature change detection system of FIG. 14. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In each drawing, the same or equivalent parts are denoted by the same reference numerals. In the following embodiments, the terms "have," "include," or "contain" also mean "consist of" or "consist of."

[0011] Embodiment 1 First, a first embodiment of the present invention will be described.

[0012] As shown in Figure 1, a three-dimensional feature change detection system 1A serving as an information processing device detects changes in three-dimensional features between a first time point t1 and a second time point t2 based on map data 2a stored in a map data storage device 2 and captured images 3a captured by a satellite and stored in a captured image data storage device 3. A three-dimensional feature is an object that is fixed to the Earth's surface, has a bottom surface area (footprint) defined in map data 2a, and has an upper surface area that is at a certain height above the Earth's surface. The first time point t1 and the second time point t2 are different times, and may occur either before or after one another.

[0013] The map data storage device 2 stores map data 2a and feature data 2b. The feature data 2b includes information such as the position, footprint, and outline of the top surface area of ​​a three-dimensional feature fixed on the Earth's surface. Furthermore, the feature data 2b includes height information 2c from the ground of the three-dimensional feature included in the map data 2a. In addition to the height dimensions of the representative point or polygon of the three-dimensional feature, if the three-dimensional feature is a building, the number of floors of the building is also included in the height information 2c. This is because once the number of floors of a building is determined, the building's height is also determined to some extent. Meanwhile, the captured image data storage device 3 stores imaging conditions 3b used when capturing the captured image 3a. The captured image 3a may be an aerial photograph taken with an aerial camera or a satellite image acquired by a sensor mounted on an artificial satellite. The imaging conditions 3b include the angle of the observation equipment relative to the ground surface (off-nadir angle), the observation azimuth angle, the azimuth direction, the range direction, the satellite's position and altitude, and the image resolution. From the imaging conditions 3b, it is possible to determine which area on the ground the captured image 3a is captured at and at what scale it is captured.

[0014] As shown in FIG. 1, the three-dimensional feature change detection system 1A includes a first feature information generation unit 11 (extraction means), an offset calculation unit 12 (calculation means), a second feature information generation unit 13, a side area estimation unit 14, a difference information generation unit 15 (comparison means), and a change determination unit 16 (determination means).

[0015] (First feature information generation unit) The first feature information generation unit 11 acquires a captured image 3a from the captured image data storage device 3. The captured image 3a is an image obtained by capturing an area including a three-dimensional feature from the sky at a first time point t1. The first feature information generation unit 11 extracts the upper surface area of ​​the three-dimensional feature based on the captured image 3a. Specifically, the first feature information generation unit 11 uses machine learning to generate a first feature group image IM1 (see FIG. 2) based on the captured image 3a, in which the upper surface area of ​​the three-dimensional feature at the first time point t1 is extracted as an area of ​​interest AA. Note that the first feature information generation unit 11 may generate polygon information indicating the position, shape, and size of the upper surface area of ​​the three-dimensional feature instead of the first feature group image IM1 (see FIG. 2). The first feature group image IM1 is first feature information indicating the position, shape, and size of the upper surface area of ​​the three-dimensional feature.

[0016] In the example shown in Fig. 2, top surface regions T1, T2, and T4 are extracted as an area of ​​interest AA in the first feature group image IM1. Because the first feature group image IM1 was captured at an angle inclined relative to the vertical direction (the off-nadir angle and observation azimuth angle described above), the top surface regions T1, T2, and T4 of the three-dimensional features are misaligned with respect to their footprints, resulting in a tilted image, as shown in Fig. 2. Furthermore, the first feature information generation unit 11 generates a first binary image BIM1 based on the first feature group image IM1, in which the top surface regions T1, T2, and T4 of the three-dimensional features have different values ​​from the other regions.

[0017] (Offset calculation section) The offset calculation unit 12 acquires feature data 2b, including height information 2c of the three-dimensional feature from the ground, from the map data storage device 2, and acquires imaging conditions 3b under which the captured image 3a was captured from the captured image data storage device 3. The acquired imaging conditions 3b include at least an observation azimuth angle, which indicates the angular deviation of the imaging direction from the vertical direction. The observation azimuth angle indicates the direction in which the top surface area of ​​the three-dimensional feature is shifted in the first feature group image IM1. Based on the acquired height information 2c and imaging conditions 3b (observation azimuth angle), the offset calculation unit 12 calculates a positional deviation vector, which indicates the deviation between the footprint on the ground surface of the three-dimensional feature and the top surface area.

[0018] FIG. 2 shows a second feature group image IM2, which is an image of the outlines of footprints of three-dimensional features located in the same region as the first feature group image IM1. According to map data 2a and feature data 2b, three three-dimensional features corresponding to footprints F1, F2, and F3 exist in this region. The offset calculation unit 12 acquires height information 2c for each of the three three-dimensional features corresponding to footprints F1, F2, and F3 located in the same region as the region shown in the first feature group image IM1 from the map data storage device 2. Furthermore, the offset calculation unit 12 acquires imaging conditions 3b from the captured image data storage device 3 when the captured image 3a was captured. Based on the acquired height information 2c and imaging conditions 3b, the offset calculation unit 12 calculates positional deviation vectors V1 to V3 indicating the deviations between the footprints F1 to F3 and the upper surface region. For the upper surface region T4 extracted from the first feature group image IM1, there is no three-dimensional feature corresponding to the feature data 2b, so height information 2c cannot be obtained and a positional deviation vector cannot be calculated.

[0019] As shown in FIG. 2, the offset calculation unit 12 determines the orientation and angle between the satellite's imaging direction and the vertical direction based on the imaging condition 3b, and calculates positional deviation vectors V1 to V3 indicating the positional deviation between the top surface area of ​​each three-dimensional feature and the footprints F1, F2, and F3 based on the orientation and angle and the height of the three-dimensional feature. The positional deviation vectors V1 to V3 are defined by pixel positions within the image. The offset calculation unit 12 calculates the scale on the Earth's surface corresponding to one pixel (distance per pixel [m]) based on the image resolution or satellite altitude included in the imaging condition 3b. The offset calculation unit 12 calculates the positional deviation vectors (pixels in the X-axis direction, pixels in the Y-axis direction) based on an XY Cartesian coordinate system that defines the pixel position determined by the scale per pixel.

[0020] (Second feature information generation unit) Returning to FIG. 1, the second feature information generation unit 13 acquires map data 2a from the map data storage device 2. The map data 2a is data showing a map at a second time point t2 that is different from the first time point t1 at which the captured image 3a was captured. Based on the acquired map data 2a, the second feature information generation unit 13 generates a map image of the same area as the area shown in the captured image 3a. The map image is an image that shows the footprint of a three-dimensional feature at the second time point t2, distinguished from other areas, and is second feature information that indicates the position, shape, and size of the top surface area of ​​the three-dimensional feature.

[0021] FIG. 2 shows an example of a map image generated by the second feature information generation unit 13. The second feature group image IM2 is an image based on map data 2a and is an image of the same area as the first feature group image IM1. That is, in this embodiment, the second feature group image IM2 is an image of a map based on map data 2a. Based on feature data 2b, the second feature information generation unit 13 shifts footprints F1, F2, and F3 by displacement vectors V1, V2, and V3 to generate a second binary image BIM2 in which values ​​of the footprints F1, F2, and F3 differ from those of other areas. Hereinafter, the footprints F1, F2, and F3 in the second binary image BIM2 are also referred to as the area of ​​interest AA. Note that the second feature information generation unit 13 may generate polygon information indicating the position, shape, and size of the footprints as the second feature information, instead of the second feature group image IM2 (see FIG. 2).

[0022] (Side area estimation part) Returning to FIG. 1 , the side area estimation unit 14 acquires information indicating the position, shape, and size of the footprint of a three-dimensional feature from the map data storage device 2. Furthermore, the side area estimation unit 14 receives the offset of the top surface area of ​​each three-dimensional feature, i.e., the displacement vector, from the offset calculation unit 12. The side area estimation unit 14 estimates the side area of ​​the three-dimensional feature based on the position, shape, and size of the footprint and the displacement vector. When the footprint is shifted by the displacement vector, the side area estimation unit 14 estimates the area of ​​the footprint that is moved, excluding the footprint after the shift, as the side area. Information on the estimated side area is sent to the first feature information generation unit 11. The first feature information generation unit 11 excludes the top surface area of ​​the extracted three-dimensional feature that is included in the side area from the area of ​​interest AA. The side area estimation unit 14 may estimate the side area of ​​the three-dimensional feature in the captured image 3a using machine learning.

[0023] 3, for example, a side area SA1 is shown surrounded by a dotted line in the first feature group image IM1, and within this area is an area AA1 extracted as an upper surface area by the first feature information generation unit 11. In this case, the first feature information generation unit 11 excludes area AA1 from the area of ​​interest AA in the first feature group image IM1, and extracts only area AA2 as the area of ​​interest AA.

[0024] (Difference information generation part) Returning to FIG. 1, the difference information generator 15 calculates difference information between information indicating the position, shape, and size of the top surface region of the three-dimensional feature obtained from the captured image 3a and information indicating the position, shape, and size of the footprint obtained from the map data 2a. For example, as shown in FIG. 2, the difference information generator 15 generates a first difference image DIM1 and a second difference image DIM2 as difference information. The first difference image DIM1 is an image generated by subtracting the second binary image BIM2 from the first binary image BIM1, and the second difference image DIM2 is an image generated by subtracting the first binary image BIM1 from the second binary image BIM2. The first difference image DIM1 and the second difference image DIM2 are generated as images in which the value of the attention area AA is 1 and the values ​​of other areas are 0.

[0025] Furthermore, as shown in Figure 4, the difference information generation unit 15 erases from the first difference image DIM1 the area of ​​interest AA where the ratio of the change in size of the first difference image DIM1 to the first binary image BIM1 (DIM1 / BIM1) is below a threshold, and erases from the second difference image DIM2 the area of ​​interest AA where the ratio of the change in size of the second difference image DIM2 to the second binary image BIM2 (DIM2 / BIM2) is below a threshold.

[0026] (Change determination section) Returning to FIG. 1 , the change determination unit 16 determines that a change has occurred in the three-dimensional feature if the top surface region and the region corresponding to the three-dimensional feature differ. The change determination unit 16 calculates the difference region between the top surface region and the region corresponding to the three-dimensional feature in the difference information generation unit 15. The change determination unit 16 determines that a change has occurred in the three-dimensional feature if it determines that the size of the difference region is equal to or greater than a predetermined threshold. For example, the change determination unit 16 determines that a change has occurred in the three-dimensional feature if the size of the attention region AA remaining in the first difference image DIM1 and the second difference image DIM2 calculated by the difference information generation unit 15 is equal to or greater than a threshold. The attention region AA is the region in which a change has occurred in the three-dimensional feature. For example, as shown in FIG. 2 , the attention region AA remaining in the first difference image DIM1 is a three-dimensional feature that existed at the first time point t1 but does not exist at the second time point t2. The area of ​​interest AA remaining in the second difference image DIM2 is a three-dimensional feature that did not exist at the first time point t1 but exists at the second time point t2. By referring to the first difference image DIM1 and the second difference image DIM2, it is possible to detect newly installed three-dimensional features and demolished three-dimensional features.

[0027] [Hardware configuration] The three-dimensional feature change detection system 1A shown in Fig. 1 is realized, for example, by a computer 20 having the hardware configuration shown in Fig. 5 executing a software program. Specifically, the computer 20 includes a CPU (Central Processing Unit) 21, which is a processor that controls the entire device, a main memory 22 such as RAM (Random Access Memory), an external memory 23 consisting of a non-volatile memory such as a flash memory or a hard disk, an operation unit 24 consisting of devices such as a keyboard and a mouse, a display 25 consisting of a display device such as a CRT (Cathode Ray Tube) or a liquid crystal monitor, a communication interface 26 that communicates data with an external computer, and an internal bus 28 that connects these elements.

[0028] The program 29 is loaded from the external memory 23 into the main memory 22 and executed by the CPU 21. This realizes the functions of the computer 20. When executing the program 29, the CPU 21 performs data communication with an external computer via the communication interface 26 as necessary. In this embodiment, the program 29 executed by the CPU 21 includes the program of the three-dimensional feature change detection system 1A.

[0029] The functions of the three-dimensional feature change detection system 1A can be implemented in a computer system consisting of one or more computers, each including one or more processors and one or more storage devices, including a non-transitory storage medium. The multiple computers realize the functions of the three-dimensional feature change detection system 1A while communicating with each other via an interconnected communication network. For example, some of the functions of the three-dimensional feature change detection system 1A may be implemented in one computer, and others may be implemented in another computer. The functions of the three-dimensional feature change detection system 1A may also be realized by a cloud computer.

[0030] (3D feature change detection processing) Next, the three-dimensional feature change detection process executed by the three-dimensional feature change detection system 1A of FIG. 1, that is, the three-dimensional feature change detection method, will be described.

[0031] 6, in the three-dimensional feature change detection system 1A, first, the first feature information generation unit 11 uses machine learning to extract the upper surface areas of the three-dimensional features at the first time point t1 as an area of ​​interest AA based on a captured image 3a captured from the sky at a first time point t1 of an area including multiple three-dimensional features (step S1; extraction step). As a result, for example, a first feature group image IM1 (information indicating the position, shape, and size of the upper surface area) shown in FIG.

[0032] Next, the offset calculation unit 12 acquires the imaging conditions 3b under which the captured image 3a was captured from the captured image data storage device 3 (step S2). Next, the offset calculation unit 12 acquires height information 2c of the three-dimensional feature (building) included in the feature data 2b from the map data storage device 2 (step S3). Next, the offset calculation unit 12 calculates a positional deviation vector (for example, V1 to V3 in FIG. 2) indicating the deviation between the footprint on the ground surface of the three-dimensional feature and the upper surface area, based on the height information 2c of the three-dimensional feature from the ground and the imaging conditions 3b under which the captured image 3a was captured (step S4; calculation step).

[0033] Next, the side area estimation unit 14 estimates the side area of ​​the three-dimensional feature (for example, side area SA1 in FIG. 3) based on the position, shape, and size of the footprint of the three-dimensional feature and the displacement vector (step S5; calculation step, exclusion step). Next, the first feature information generation unit 11 excludes areas included in the side area from the top area, for example, as shown in FIG. 3 (step S6; calculation step, exclusion step). For example, in the example shown in FIG. 3, area AA1 is excluded from side area SA1.

[0034] 2, the first feature information generation unit 11 generates a first binary image BIM1 (top surface area information) in which values ​​differ between the area of ​​interest AA and other areas based on the first feature group image IM1, and the second feature information generation unit 13 generates a second binary image BIM2 (footprint information) in which values ​​differ between the area of ​​interest AA and other areas based on the map data 2a and the feature data 2b (step S7). Here, the second feature information generation unit 13 generates the second binary image BIM2 (footprint information) in which the deviation between the footprint and the top surface area is canceled.

[0035] Next, the difference information generation unit 15 generates a first difference image DIM1 by subtracting the second binary image BIM2 (footprint information) from the first binary image BIM1 (top surface area information) and calculates the difference between the two, and also generates a second difference image DIM2 (difference information) by subtracting the first binary image BIM1 (top surface area information) from the second binary image BIM2 (footprint information) and calculates the difference between the two, and compares the top surface area extracted from the captured image 3a captured at the first time point t1 with the area corresponding to the three-dimensional feature at the second time point t2 (step S8; comparison step, difference calculation step). In this comparison step, the difference information generation unit 15, as shown in Figure 4, erases from the first difference image DIM1 the area of ​​interest AA where the ratio of the change in size of the first difference image DIM1 to the first binary image BIM1 is below a threshold, and erases from the second difference image DIM2 the area of ​​interest AA where the ratio of the change in size of the second difference image DIM2 to the second binary image BIM2 is below a threshold.

[0036] Next, in the comparison step (step S8), if the upper surface area and the area of ​​the three-dimensional feature, i.e., the footprint, are different, that is, if it is determined that the difference is equal to or greater than a predetermined threshold, the change determination unit 16 determines that there has been a change in the three-dimensional feature (step S9; determination step).

[0037] In this embodiment, the area of ​​the three-dimensional feature at a position corresponding to the footprint extracted from the map data 2a is shifted, and this area is compared with the top surface area of ​​the three-dimensional feature in the captured image 3a. This is because the area linked to the height information 2c is the area of ​​the three-dimensional feature based on the feature data 2b, and it is unclear how the top surface area extracted from the captured image 3a is associated with the height information 2c.

[0038] In this embodiment, the second feature information generation unit 13 generates a second binary image BIM2 and shifts the extracted footprint by the displacement vector. However, the second feature information generation unit 13 may first generate information indicating the position, shape, and size of the footprint, shift the position of the footprint by the displacement vector, and generate a second binary image BIM2 indicating the shifted footprint.

[0039] Alternatively, the second feature information generation unit 13 may generate information indicating the position, shape, and size of the footprint without generating the second binary image BIM2, and may simply shift the information indicating that position by the displacement vector. In this case, the first feature information generation unit 11 may generate polygon information indicating the position, shape, and size of the upper surface area of ​​the three-dimensional feature, rather than the first feature group image IM1 (see FIG. 2). The difference information generation unit 15 may calculate difference information (information indicating the position, shape, and size of the difference area) between the information indicating the position, shape, and size of the footprint shifted by the displacement vector and the polygon information indicating the position, shape, and size of the upper surface area of ​​the three-dimensional feature. The change determination unit 16 determines that a change has occurred in the three-dimensional feature based on this difference information.

[0040] Embodiment 2 Next, a second embodiment of the present invention will be described. In the first embodiment, the area of ​​a three-dimensional feature located at a position corresponding to a footprint extracted from map data 2a is shifted to generate a first difference image DIM1 and a second difference image DIM2 between that area and the top surface area of ​​the three-dimensional feature in the captured image 3a. In the present embodiment, the top surface area of ​​the three-dimensional feature in the captured image 3a is shifted to generate a first difference image DIM1 and a second difference image DIM2 between the area of ​​the three-dimensional feature located at a position corresponding to the footprint. Note that detailed description of the configuration and operation of three-dimensional feature change detection system 1B according to the second embodiment that are the same as those of three-dimensional feature change detection system 1A according to the first embodiment will be omitted. The same applies to the third and fourth embodiments described below.

[0041] As shown in Fig. 7, the three-dimensional feature change detection system 1B according to this embodiment includes a first feature information generation unit 31 instead of the first feature information generation unit 11, an offset calculation unit 32 instead of the offset calculation unit 12, and a second feature information generation unit 33 instead of the second feature information generation unit 13. The first feature information generation unit 31 is the same as the first feature information generation unit 11 in that it generates, based on the captured image 3a, information indicating the top surface region of a three-dimensional feature, i.e., a first binary image BIM1 in which the values ​​for the top surface region and other regions are different.

[0042] The offset calculation unit 32 is similar to the offset calculation unit 12 in that it calculates the deviation between the footprint on the ground surface of the three-dimensional feature and the upper surface area based on height information 2c of the three-dimensional feature from the ground and imaging conditions 3b when the captured image 3a was captured. As shown in FIG. 8, for example, the offset calculation unit 32 shifts the footprints F1, F2, and F3 by the calculated positional deviation vectors V1, V2, and V3, and performs an overlap determination with the upper surface area in the first binary image BIM1 generated by the first feature information generation unit 11. In the overlap determination, if the ratio of the overlapping portion to the entire image is equal to or greater than a threshold, it is determined that there is overlap. The offset calculation unit 32 associates the footprints F1 to F3 with the upper surface area in the overlapping area between the footprints and the upper surface area. As a result, for example, the footprint F1 is associated with the upper surface area T1, and the footprint F2 is associated with the upper surface area T2.

[0043] As shown in FIG. 9, the first feature information generation unit 31 generates information indicating the upper surface regions, i.e., the first binary image BIM1 (FIG. 2), by shifting the upper surface regions T1 and T2 by displacement vectors V1 and V2 to cancel the misalignment between the footprints F1 and F2 associated by the offset calculation unit 32 and the upper surface regions T1 and T2. Meanwhile, the second feature information generation unit 33 generates information indicating the footprints, i.e., the second binary image BIM2 in which the values ​​of the footprints F1, F2, and F3 differ from those of other regions, based on the second feature group image IM2, without shifting the footprints F1, F2, and F3. The processing after generating the first binary image BIM1 and the second binary image BIM2 is the same as in the first embodiment. The arrows in FIG. 9 indicate the direction in which the upper surface regions are shifted.

[0044] Next, we will explain the three-dimensional feature change detection process executed by the three-dimensional feature change detection system 1B of Fig. 7. As shown in Fig. 10, the three-dimensional feature change detection process of this embodiment differs from the above-mentioned embodiments in that step S10 is performed between steps S6 and S7. In step S10, the offset calculation unit 32 shifts the footprints F1, F2, and F3 by the calculated positional deviation vectors V1, V2, and V3, as shown in Fig. 8, for example, and determines whether the footprints F1, F2, and F3 overlap with the upper surface region in the first binary image BIM1 generated by the first feature information generation unit 11. In the overlapping region between the footprints and the upper surface region, the footprints F1 to F3 are associated with the upper surface region (step S10; matching step).

[0045] Then, the first feature information generation unit 31 shifts the upper surface areas T1 and T2 by the positional deviation vectors V1 and V2 in accordance with the relationship between the footprints F1 and F2 and the upper surface areas T1 and T2 associated by the offset calculation unit 32, as shown in, for example, FIG. 9, to generate a first binary image BIM1 (FIG. 2) (step S7).

[0046] In this way, the top surface area can be shifted toward the footprint by the displacement vector, but in this case, it is necessary to shift the footprint toward the top surface area by the displacement vector to associate the footprint with the top surface area.

[0047] The first feature information generation unit 31 may generate information indicating the position, shape, and size of the upper surface area, and may shift the position of the upper surface area by a displacement vector to generate a first binary image BIM1 indicating the displaced footprint.

[0048] Alternatively, the first feature information generator 31 may generate information indicating the position, shape, and size of the top surface region of the three-dimensional feature without generating the first binary image BIM1, and may simply shift the information indicating that position by the displacement vector. In this case, the second feature information generator 33 may generate information indicating the position, shape, and size of the footprint of the three-dimensional feature, rather than the second binary image BIM2 (see FIG. 2). The difference information generator 15 may calculate difference information (information indicating the position, shape, and size of the difference region) between the information indicating the position, shape, and size of the top surface region shifted by the displacement vector and polygon information indicating the position, shape, and size of the footprint of the three-dimensional feature. The change determination unit 16 determines that a change has occurred in the three-dimensional feature based on this difference information.

[0049] Embodiment 3 Next, a third embodiment of the present invention will be described. In the first and second embodiments described above, the upper surface area of ​​a three-dimensional feature shown in the captured image 3a at the first time point t1 is compared with the area of ​​the three-dimensional feature at a position corresponding to the footprint, and a change therebetween is detected. In this embodiment, the upper surface area of ​​the three-dimensional feature shown in the captured image 3a at the first time point t1 is compared with the upper surface area of ​​the three-dimensional feature shown in the captured image 3a captured at the second time point t2, and a change therebetween is detected.

[0050] 11, three-dimensional feature change detection system 1C according to this embodiment differs from three-dimensional feature change detection system 1A in that it includes a first feature information generation unit 41, an offset calculation unit 42, a second feature information generation unit 43, a side area estimation unit 44, and a difference information generation unit 45, instead of first feature information generation unit 11, offset calculation unit 12, second feature information generation unit 13, side area estimation unit 14, and difference information generation unit 15. Furthermore, this embodiment does not need to include a map data storage device 2.

[0051] The first feature information generation unit 41 acquires a captured image 3a captured at a first time point t1 from the captured image data storage device 3. Based on this captured image 3a, the first feature information generation unit 41 extracts the top surface areas of three-dimensional features and generates first top surface area information. As shown in Fig. 12, the image from which the top surface areas have been extracted is designated as a first feature group image IM1, and the extracted top surface area is designated as T1(AA).

[0052] The second feature information generation unit 43 acquires a captured image 3a captured at a second time point t2 from the captured image data storage device 3. Based on this captured image 3a, the second feature information generation unit 43 extracts the top surface area of ​​the three-dimensional feature and generates second top surface area information. As shown in FIG. 12, the image from which the top surface area has been extracted is designated as a second feature group image IM2, and the extracted top surface area is designated as T2(AA). The top surface area T1(AA) and the top surface area T2(AA) are the top surface areas of the same three-dimensional feature.

[0053] The offset calculation unit 42 performs machine learning on the first feature group image IM1 to estimate a displacement vector V1 indicating the displacement between the footprint FP of the three-dimensional feature and the upper surface area T1(AA). The offset calculation unit 42 also performs machine learning on the second feature group image IM2 to estimate a displacement vector V2 indicating the displacement between the footprint FP of the three-dimensional feature and the upper surface area T2(AA). That is, while the displacement vector is calculated based on the height information 2c and the imaging conditions 3b in the first and second embodiments, in this embodiment the offset calculation unit 42 estimates the displacement vector between the upper surface area and the footprint based on the captured image 3a. Note that in FIG. 12, the direction from the upper surface area to the footprint is shown as the direction of the vector.

[0054] The first feature information generation unit 41 shifts the upper surface region T1 (AA) onto the footprint FP based on the positional deviation vector V1 to generate a first binary image BIM1 (first upper surface region information). The second feature information generation unit 43 shifts the upper surface region T2 (AA) onto the footprint FP based on the positional deviation vector V2 to generate a second binary image BIM2 (second upper surface region information).

[0055] The side area estimation unit 44 acquires the captured image 3a from the captured image data storage device 3 and estimates the top surface area of ​​the three-dimensional feature in the captured image 3a through machine learning. As shown in FIG. 12, the side area estimation unit 44 estimates the side surface area SA1 of the three-dimensional feature based on the estimated top surface area T1 and the displacement vector V1 calculated by the offset calculation unit 42. The side area SA1 is the area into which the top surface area T1 moves when the top surface area is shifted by the displacement vector V1, excluding the top surface area T1 after the shift. The first feature information generation unit 41 excludes the top surface area included in the side area SA1 from the top surface area of ​​the extracted three-dimensional feature. As shown in FIG. 12, the side area estimation unit 44 estimates the side surface area SA2 of the three-dimensional feature based on the estimated top surface area T2 and the displacement vector V2 calculated by the offset calculation unit 42. The region into which the top surface region T2 moves when the top surface region T2 is shifted by the positional deviation vector V2, excluding the moved top surface region T2, is estimated as the side surface region SA2. The second feature information generation unit 43 excludes the top surface region included in the side surface region SA2 from the top surface regions of the extracted three-dimensional feature.

[0056] The difference information generating unit 45 generates difference information between the first top surface area information and the second top surface area information. The change determining unit 16 determines that a change has occurred in the three-dimensional feature based on this difference information.

[0057] Next, we will explain the three-dimensional feature change detection processing executed by the three-dimensional feature change detection system 1C of Fig. 11. As shown in Fig. 13, the three-dimensional feature change detection processing of this embodiment differs from that of Embodiment 1 above in that it performs step S11 instead of steps S2 to S4, and performs step S12 instead of step S7. However, in step S1, the first feature information generation unit 41 extracts an upper surface area, and the second feature information generation unit 43 extracts an upper surface area, and in steps S5 and S6, the first feature information generation unit 41 excludes a side area, and the second feature information generation unit 43 excludes a side area.

[0058] In step S11, the offset calculation unit 42 performs machine learning on the first feature group image IM1 to estimate a displacement vector V1 indicating the displacement between the footprint FP of the three-dimensional feature and the top surface region T1(AA), and performs machine learning on the second feature group image IM2 to estimate a displacement vector V2 indicating the displacement between the footprint FP of the three-dimensional feature and the top surface region T2(AA).Furthermore, in step S12, the first feature information generation unit 41 shifts the top surface region T1(AA) onto the footprint FP based on the displacement vector V1 to generate a first binary image BIM1 (first top surface region information), and the second feature information generation unit 43 shifts the top surface region T2(AA) onto the footprint FP based on the displacement vector V2 to generate a second binary image BIM2 (second top surface region information).

[0059] In step S8, the difference information generating unit 45 generates difference information between the first upper surface region information and the second upper surface region information. The method of generating the difference information and the process of the change determining unit 16 using the difference information are the same as those in the first and second embodiments.

[0060] In this way, by shifting the upper surface area T2(AA) onto the footprint FP, it is possible to prevent overlap with other three-dimensional features, and to prevent erroneous detection of changes in three-dimensional features.

[0061] Embodiment 4 Next, a fourth embodiment of the present invention will be described. This embodiment is the same as the third embodiment in that the upper surface area of ​​a three-dimensional object shown in the captured image 3a at the first time point t1 is compared with the upper surface area extracted from the captured image 3a captured at the second time point t2 to detect any change therebetween.

[0062] As shown in FIG. 14 , three-dimensional feature change detection system 1D according to this embodiment differs from three-dimensional feature change detection system 1C according to the third embodiment in that it includes a first feature information generation unit 51, an offset calculation unit 52, a second feature information generation unit 53, and a side area estimation unit 54 instead of first feature information generation unit 41, offset calculation unit 42, second feature information generation unit 43, and side area estimation unit 44, and in that it includes a map data storage device 2.

[0063] The first feature information generation unit 51 acquires a captured image 3a captured at a first time point t1 from the captured image data storage device 3. The first feature information generation unit 51 extracts the upper surface areas of three-dimensional features based on this captured image 3a. As shown in Fig. 15(A), the image from which the upper surface area T1 (AA) has been extracted is designated as a first feature group image IM1 (first upper surface area information).

[0064] The second feature information generation unit 53 obtains the captured image 3a captured at the second time point t2 from the captured image data storage device 3. The second feature information generation unit 53 extracts the top surface area of ​​the three-dimensional feature based on the captured image 3a. As shown in FIG. 15(B), the image from which the top surface area T2(AA) has been extracted is defined as the second feature group image IM2 (second top surface area information). The top surface area T1(AA) and the top surface area T2(AA) are the top surface areas of the same three-dimensional feature.

[0065] The offset calculation unit 52 calculates a position vector V1 indicating the deviation between the footprint on the ground surface of the three-dimensional feature and the upper surface region based on height information 2c of the three-dimensional feature from the ground and imaging conditions 3b when captured image 3a at first time point t1. As shown in FIG. 15(A), for example, the offset calculation unit 52 generates an image MD1 in which the footprint F1 is shifted by the calculated position deviation vector V1, and determines whether the footprint F1 in image MD1 overlaps with the upper surface region T1(AA) of the first feature group image IM1. If the overlap ratio between the footprint F1 and the upper surface region T1(AA) is equal to or greater than a threshold, the offset calculation unit 52 associates the footprint F1 with the upper surface region T1(AA).

[0066] The offset calculation unit 52 calculates a position vector V2 indicating the deviation between the footprint on the ground surface of the three-dimensional feature and the upper surface region, based on height information 2c of the three-dimensional feature from the ground and imaging conditions 3b when captured image 3a at second time point t2. As shown in FIG. 15(B), for example, the offset calculation unit 52 shifts the footprint F2 by the calculated position deviation vector V2 and determines whether it overlaps with the upper surface region in the first binary image BIM1 generated by the first feature information generation unit 51. For example, the offset calculation unit 52 associates the footprint F2 with the upper surface region T2(AA) in the overlapping region between the footprint F1 and the upper surface region T2(AA).

[0067] 15(C), for example, the first feature information generation unit 51 generates a first binary image BIM1 (see FIG. 2) by shifting the upper surface region T1 by a positional deviation vector V1 in accordance with the relationship between the footprint F1 and the upper surface region T1 associated by the offset calculation unit 52. In FIG. 15(C), the direction from the upper surface region T1 toward the footprint F1, i.e., the direction in which the upper surface region T1 is shifted, is shown as the direction of the vector.

[0068] 15(C), the second feature information generation unit 53 generates a second binary image BIM2 (see FIG. 2) by shifting the upper surface region T2 by the positional deviation vector V2 in accordance with the relationship between the footprint F2 and the upper surface region T2 associated by the offset calculation unit 52. The subsequent flow is the same as in the first and second embodiments.

[0069] The operation of the side area estimation unit 54 is the same as that of the side area estimation unit 14 in the first embodiment. The side area estimation unit 54 estimates the side area of ​​a three-dimensional feature based on the position, shape, and size of the footprint acquired from the map data storage device 2 and the displacement vector V1 calculated by the offset calculation unit 52. The side area estimation unit 54 also estimates the side area of ​​a three-dimensional feature based on the position, shape, and size of the footprint acquired from the map data storage device 2 and the displacement vector V2 calculated by the offset calculation unit 52. Information about the side area estimated from the displacement vector V1 is output to the first feature information generation unit 51, and information about the side area estimated from the displacement vector V2 is output to the second feature information generation unit 53, and these are used to exclude the respective side areas.

[0070] Next, the three-dimensional feature change detection processing executed by the three-dimensional feature change detection system 1D of Fig. 14 will be described with reference to Fig. 16. Here, differences from the three-dimensional feature change detection processing of the three-dimensional feature change detection system 1C according to the third embodiment shown in Fig. 16 will be described.

[0071] In this three-dimensional feature change detection processing, step S10 is performed between step S11 and step S5. In step S10, the offset calculation unit 52 shifts the footprint F1 by the calculated positional deviation vector V1, as shown in Fig. 15(A), for example, and determines whether the footprint F1 overlaps with the upper surface region extracted by the first feature information generation unit 51. In the overlapping region between the footprint F1 and the upper surface region, the offset calculation unit 52 associates the footprint F1 with the upper surface region T1 (matching step). Similarly, as shown in Fig. 15(B), for example, the offset calculation unit 52 shifts the footprint F2 by the calculated positional deviation vector V2, and determines whether the footprint F2 overlaps with the upper surface region extracted by the second feature information generation unit 53. In the overlapping region between the footprint F2 and the upper surface region, the offset calculation unit 52 associates the footprint F2 with the upper surface region T2 (matching step).

[0072] 15(C), the first feature information generation unit 51 shifts the top surface region T1 by the positional shift vector V1 to generate first top surface region information, and the second feature information generation unit 53 shifts the top surface region T2 by the positional shift vector V2 to generate second top surface region information (step S12). The subsequent processing is the same as that of the three-dimensional feature change detection system 1C according to the third embodiment.

[0073] (summary) (1) As explained in detail above, the method for detecting changes in three-dimensional features according to this embodiment can cancel out the positional deviation between the top surface area and the footprint caused by the collapse of a three-dimensional feature in the captured image 3a taken at the first time point t1, thereby enabling changes in three-dimensional features to be detected with high accuracy.

[0074] (2) According to the three-dimensional feature change detection method of the above embodiment, the attention area AA included in the side area SA1 of the three-dimensional feature in the captured image 3a is excluded from the top surface area, thereby preventing the side area of ​​the three-dimensional feature from being erroneously detected as the top surface area.

[0075] (3) In the three-dimensional feature change detection method according to the above embodiment, in the comparison step, before shifting the top surface area toward the footprint, the footprint is shifted toward the top surface area by the position vector, and the footprint and the top surface area are associated. In this way, the footprint and the top surface area of ​​different three-dimensional features can be prevented from being treated as the same three-dimensional feature, making it possible to detect changes in three-dimensional features with high accuracy.

[0076] The height information is not limited to that in the above embodiment. Any information indicating heights from the ground surface associated with multiple points on the ground can be used. For example, height information obtained by airborne laser surveying may be combined with map data and used as the height information.

[0077] In the above embodiment, the three-dimensional features for which changes are to be detected are primarily buildings such as buildings, but the present invention is not limited to this. For example, the three-dimensional features may be structures such as highways. The three-dimensional features may also be naturally formed objects.

[0078] In the above embodiment, the upper surface area of ​​a three-dimensional feature is extracted by machine learning, but this is not limiting and other image processing may be used to extract the upper surface area of ​​a three-dimensional feature.

[0079] In the above embodiment, the captured image 3a is taken by a satellite, but this is not limiting, and the captured image 3a may be, for example, an aerial photograph taken by an airplane.

[0080] The hardware and software configurations of the three-dimensional feature change detection systems 1A to 1D are merely examples, and can be changed and modified as desired.

[0081] The core processing section of the computer 20, which is composed of the CPU 21, main memory 22, external memory 23, operation unit 24, display 25, communication interface 26, internal bus 28, etc., can be realized using an ordinary computer system rather than a dedicated system. For example, a computer program for executing the above operations may be stored and distributed on a computer-readable recording medium (such as a flexible disk, CD-ROM, or DVD-ROM), and the three-dimensional feature change detection systems 1A-1D that execute the above processing may be configured by installing the computer program on a computer. Alternatively, the three-dimensional feature change detection systems 1A-1D may be configured by storing the computer program in a storage device of a server device on a communication network such as the Internet, and downloading the program into an ordinary computer system.

[0082] When the functions of the three-dimensional feature change detection systems 1A to 1D are realized by sharing the functions between an OS (operating system) and an application program, or by cooperation between the OS and the application program, only the application program portion may be stored in a recording medium or storage device.

[0083] It is also possible to superimpose a computer program on a carrier wave and distribute it over a communications network. For example, the computer program may be posted on a bulletin board system (BBS) on the communications network and distributed over the network. The computer program may then be started and executed under the control of an operating system in the same way as any other application program, thereby enabling the above-mentioned processing to be performed.

[0084] This invention allows various embodiments and modifications without departing from the broad spirit and scope of this invention. Furthermore, the above-described embodiments are intended to explain this invention and do not limit the scope of this invention. That is, the scope of this invention is defined by the claims, not the embodiments. Various modifications made within the scope of the claims and the meaning of the invention equivalent thereto are considered to be within the scope of this invention. [Industrial Applicability]

[0085] The present invention can be applied to detect changes in three-dimensional features. [Explanation of symbols]

[0086] 1A, 1B, 1C, 1D: Three-dimensional feature change detection system (information processing device), 2: map data storage device, 2a: map data, 2b: feature data, 2c: height information, 3: captured image data storage device, 3a: captured image, 3b: imaging conditions, 11: first feature information generation unit (extraction means), 12: offset calculation unit (calculation means), 13: second feature information generation unit, 14: side area estimation unit, 15: difference information generation unit (comparison means), 16: change determination unit (determination means), 20: computer, 21: CPU, 22: main memory, 23: external memory, 24: operation unit, 25: display, 26: communication interface, 28: internal bus, 29: program, 31: first feature information generation unit, 32: offset calculation unit, 33: second feature information generation unit, 41: first feature information generation unit, 42: offset calculation unit, 43: second feature information generation unit, 44 Side area estimation unit, 45 Difference information generation unit, 51 First feature information generation unit, 52 Offset calculation unit, 53 Second feature information generation unit, 54 Side area estimation unit, AA Area of ​​interest, AA1, AA2 Area, IM1 First feature group image, IM2 Second feature group image, BIM1 First binary image, BIM2 Second binary image, DIM1 First difference image, DIM2 Second difference image, MD1, MD2 Image, V1, V2, V3 Positional deviation vector, SA1, SA2 Side area, T1, T2, T4 Top area, F1, F2, F3, FP Footprint

Claims

1. A three-dimensional feature change detection method executed by an information processing device, comprising: an extraction step of extracting an upper surface area of ​​the three-dimensional feature based on a captured image obtained by capturing an image of an area including the three-dimensional feature from above; a calculation step of calculating a deviation between the footprint of the three-dimensional feature on the ground surface and the upper surface area based on height information of the three-dimensional feature from the ground and imaging conditions when the captured image was taken; a comparison step of comparing the top surface area of ​​the three-dimensional feature extracted from the captured image taken at a first time point with an area corresponding to the three-dimensional feature at a second time point different from the first time point, with the deviation between the footprint and the top surface area canceled; a determining step of determining that a change has occurred in the three-dimensional feature when the comparison step shows that the upper surface area and the area corresponding to the three-dimensional feature are different; having A method for detecting changes in three-dimensional features.

2. the calculating step includes an excluding step of excluding a side surface region of the three-dimensional object in the captured image from the top surface region. The method for detecting changes in three-dimensional features according to claim 1 .

3. the comparing step includes a difference calculating step of calculating a difference between the upper surface area and an area corresponding to the three-dimensional feature, In the determining step, if it is determined that the difference calculated in the difference calculating step is equal to or greater than a predetermined threshold, it is determined that a change has occurred in the three-dimensional feature.

3. A method for detecting changes in three-dimensional features according to claim 1 or 2.

4. On the computer, an extraction step of extracting an upper surface area of ​​the three-dimensional feature based on a captured image obtained by capturing an image of an area including the three-dimensional feature from above; a calculation step of calculating a deviation between the footprint of the three-dimensional feature on the ground surface and the upper surface area based on height information of the three-dimensional feature from the ground and imaging conditions when the captured image was taken; a comparison step of comparing the top surface area of ​​the three-dimensional feature extracted from the captured image taken at a first time point with an area corresponding to the three-dimensional feature at a second time point different from the first time point, with the deviation between the footprint and the top surface area canceled; a determining step of determining that a change has occurred in the three-dimensional feature when the comparison step shows that the upper surface area and the area corresponding to the three-dimensional feature are different; Execute program.

Images