Building image clustering device and building footprint generation device

Abstract

From a set of images of numerous buildings, each taken from multiple viewpoints, the image data of the same building is appropriately clustered into a single cluster. [Solution] The building image clustering device 1 receives an image data receiving unit 105 that takes images of two or more buildings from multiple locations, a feature vector acquisition unit 106 that acquires a feature vector for each image data by using a visual language model on the image data group, and a rastering unit 107 clusters the image data group into clusters based on the feature vectors, with groups of image data that are estimated to have been taken of the same building being treated as one cluster.

Description

Technical Field

[0001] The present invention relates to an apparatus for clustering a group of image data obtained by photographing a plurality of buildings from various positions, and an apparatus for generating a footprint of a building on a map using the apparatus.

Background Art

[0002] OpenStreetMap (hereinafter referred to as "OSM") is known as map data that can be freely used regardless of commercial or non-commercial use. Since OSM can be used for free, it is also widely used in software that uses map data. However, since OSM is created and edited by an unspecified number of participants, there may be cases where buildings that should be on the map data are not registered depending on the area. In such a case, when using map data including the area in software, software developers need to complement the footprint indicating the position and range of the building on the map data. On the other hand, the following patent document shows a method of obtaining 3D data of a building using a group of image data obtained by photographing the building from a plurality of positions. If 3D data of a building is obtained, the footprint of the building can be calculated by projecting this onto a plane. Also, each of the above documents shows that, as image data of a photographed building, a group of image data photographed by a drive recorder is used. Therefore, it is considered that the footprint of a building can be obtained by using the method described in the above patent document using a group of images photographed by a drive recorder.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, when generating building footprints using images captured by a dashcam, it is first necessary to cluster the image data of the same building into a single cluster. However, since dashcam videos capture 500 to 1000 buildings per minute, processing this manually is not practical. Furthermore, while the aforementioned patent document shows how to obtain SfM point clouds of landscape images from a dashcam, a method of extracting image data sets with common feature points from 3D data of parts that make up a building could also be considered. However, SfM point clouds are noisy and sparse, making it difficult to accurately cluster image data sets. In view of these problems, the present invention aims to appropriately cluster image data of the same building from a group of images of multiple buildings taken from multiple viewpoints, and to enable the automatic generation of building footprints on map data from the image data obtained by clustering. [Means for solving the problem]

[0005] To solve the above problems, the present invention has the following configuration. (1) A building image clustering device comprising: an image data receiving unit that receives input of image data sets of two or more buildings each taken from multiple locations; a feature vector acquisition unit that acquires a feature vector for each image data set by using a visual language model on the image data set; and a clustering unit that clusters the image data sets, with groups of image data that are estimated to have been taken of the same building considered as one cluster, based on the feature vectors. Various clustering methods can be employed as long as they use feature vectors.

[0006] (2) The building image clustering device includes a landscape image receiving unit that receives landscape images including the building taken from multiple locations, and a building image extraction unit that generates an image of the building extracted from the landscape image, and the image data receiving unit receives multiple image data generated by the building image extraction unit as input as the image data group. The image of the building extracted includes an image that has undergone additional processing such as distortion correction and removal of unwanted objects.

[0007] (3) The building image clustering device includes a reference image data acquisition unit that acquires a group of image data from the group of image data that is deemed to have captured the same building as a reference image data group, and a clustering adjustment unit that adjusts the clustering unit so that when clustered by the clustering unit, there are clusters that match or approximate the reference image data group.

[0008] (4) The building image clustering device includes: a building location-related information receiving unit that receives building location-related information which is information that can identify the location of a building displayed in each image of the image data group; a building footprint generation unit that generates a building footprint as a building footprint based on a 2D point cloud obtained by projecting a 3D point cloud obtained by 3D reconstruction of a building estimated to be the subject of photography based on the image data group onto a horizontal plane, based on the building location-related information and the image data group belonging to the cluster clustered by the clustering unit; and a cluster integration unit that determines the degree of overlap between the building footprints and integrates the image data groups that were the source of the building footprints whose overlap exceeds a predetermined standard into the same cluster. Note that the image data group used for 3D reconstruction does not need to be exactly the same as the image data received by the image data receiving unit, and may also include images that have been processed from the image data group received by the image data receiving unit. The same applies below.

[0009] (5) The building image clustering device includes: a building location-related information receiving unit that receives building location-related information which is information that can identify the location of a building displayed in each image of the image data group; a building footprint generation unit that generates a building footprint as a building footprint based on a 2D point cloud obtained by projecting a 3D point cloud obtained by 3D reconstruction of a building estimated to be the subject of photography based on the image data group onto a horizontal plane, based on the building location-related information and the image data group belonging to the cluster clustered by the clustering unit; a single-image footprint generation unit that generates a building footprint as a single-image footprint based on a 2D point cloud obtained by projecting a 3D point cloud obtained by 3D reconstruction of a building displayed in the image from a monocular viewpoint onto a horizontal plane, based on the building location-related information and each image data belonging to each of the clusters; and an image data exclusion unit that determines the degree of overlap between the single-image footprint belonging to the cluster and the building footprint obtained from the image data group belonging to the cluster, and excludes from the cluster image data that generated a single-image footprint that does not overlap or whose overlap is below a predetermined standard.

[0010] (6) The building image clustering device has an image data transfer unit that determines whether there are any building footprints whose single image footprints generated from the image data excluded by the image data exclusion unit overlap by a predetermined standard or more, and if there are any, transfers the excluded image data to the cluster to which the group of image data that generated the building footprints belong.

[0011] (7) The building image clustering device has an additional cluster generation unit which generates a new cluster of two or more image data groups that have been excluded by the image data exclusion unit, which cannot be transferred by the image data transfer unit, and whose image footprints overlap by a predetermined standard or more.

[0012] (8) The building image clustering device has a representative image extraction unit that extracts image data that generates a feature vector closest to the average vector of all feature vectors of the image data group constituting the final cluster as representative image data of the cluster.

[0013] (9) A building footprint generation device comprising: a building image clustering device as described in (1) to (4) above; a building location-related information receiving unit that receives building location-related information which is information that can identify the location of a building displayed in each image of the image data group; and a building footprint generation unit that generates a building footprint based on the building location-related information and the image data group belonging to each cluster clustered by the building clustering device.

[0014] (10) The building footprint generation device includes a map data receiving unit that receives map data including a known building footprint which is the footprint of a known building that encompasses the area in which the building is located, and a footprint integrating unit that determines whether there are any building footprints generated by the footprint generation unit that overlap with the known building footprint of the map data by a predetermined standard or more, and if there are, integrates the building footprint that is determined to exist with the known building footprint that overlaps it.

[0015] (11) A program that enables a computer to perform the function of the building image clustering device. [Effects of the Invention]

[0016] With the configuration described above, the building image clustering device according to the present invention can appropriately cluster groups of image data of the same building from a group of image data of two or more buildings taken from multiple locations. Furthermore, the building footprint generation device according to the present invention can automatically obtain the footprint of each building from a group of image data of two or more buildings taken from multiple locations.

Brief Description of the Drawings

[0017] [Figure 1] It is a diagram schematically showing a network constituting the entire system according to an embodiment. [Figure 2] It is a block diagram showing an overview of the hardware configuration of a computer according to an embodiment. [Figure 3] It is a functional block diagram schematically showing the functions of a building footprint generation device according to Embodiment 1. [Figure 4] (a) is a diagram showing an example of a landscape image, and (b) is a diagram showing an example of image data obtained by cutting out the building part from the landscape image. [Figure 5] (a) is a diagram schematically showing a group of image data, and (b) is a diagram schematically showing the clustered group of image data. [Figure 6] (a) is a diagram showing an example of OSM, and (b) is a diagram showing an example of a landscape image. [Figure 7] It is a diagram schematically showing the procedure for generating a building footprint. [Figure 8] (a) is a diagram showing an example of a building footprint, and (b) is a diagram showing an example of a building footprint and a single-image footprint. [Figure 9] It is a diagram schematically showing the procedure for generating a single-image footprint. [Figure 10] It is a flowchart showing the first half of the operations of a building footprint generation device according to Embodiment 1. [Figure 11] It is a flowchart showing the second half of the operations of a building footprint generation device according to Embodiment 1. [Figure 12] It is a functional block diagram schematically showing the functions of a building footprint generation device according to Embodiment 2. [Figure 13] It is a flowchart showing the first half of the operations of a building footprint generation device according to Embodiment 2.

Modes for Carrying Out the Invention

[0018] The embodiments of the present invention will be described below with reference to the drawings. Note that the embodiments described below are all specific examples of the invention. The numerical values, components, etc., shown in the embodiments below are examples and are not intended to limit the present invention. Furthermore, components in the embodiments below that are not described in the independent claim indicating the highest-level concept will be described as optional components. Also, the contents of each embodiment can be combined. Furthermore, for the sake of explanation, some components may be omitted from the drawings. Also, parts common to one or more embodiments may be denoted by the same reference numerals, and their descriptions may be omitted.

[0019] [Embodiment 1] Figure 1 schematically shows the network that constitutes the entire system including the building footprint generation device 1 according to this embodiment. In the communication network C shown in Figure 1, the computer A that constitutes the building footprint generation device 1 and the server M, which consists of a computer that has an OSM database as map data, are connected to the Internet as a communication network.

[0020] Figure 2 shows a block diagram illustrating the hardware configuration of computer A. Computer A is a general-purpose computer, specifically consisting of a CPU 11 for arithmetic processing, a GPU 12 that works in parallel with the CPU 11 to process large amounts of data such as image data, RAM 13 which serves as the CPU 11's workspace, ROM 14 for storing basic programs and data, storage 15 such as a hard disk or SSD for storing data and programs, input / output devices 17 including input devices such as a keyboard and mouse connected via an interface 16, output devices such as a monitor and speakers, and input / output devices such as a DVD drive, and a network interface 18 such as a router for connecting to the internet. The building footprint generation device 1 is realized by incorporating a program into computer A that performs the operations described later.

[0021] Figure 3 shows a schematic functional block diagram illustrating the functions of the building footprint generation device 1, which is formed by incorporating a predetermined program into computer A. The building footprint generation device 1 generates a building footprint based on images from a drive recorder. Furthermore, in the process of generating the building footprint, the building footprint generation device 1 extracts images of buildings from the drive recorder image data and clusters the extracted image data group into clusters, with groups of image data presumed to have captured the same building forming a single cluster. In this respect, the building footprint generation device 1 also functions as a building image clustering device 10.

[0022] The building footprint generation device 1 has the following functions: landscape image receiving unit 101, building location-related information receiving unit 102, building image cropping unit 103, building location calculation unit 104, image data receiving unit 105, feature vector acquisition unit 106, clustering unit 107, map data receiving unit 108, building footprint generation unit 109, footprint integration unit 110, cluster reorganization unit 111, footprint centroid extraction unit 112, and representative image extraction unit 113. The building footprint generation device 1 exchanges data with AI models and software installed on computer A via API linkage.

[0023] The landscape image receiving unit 101 accepts landscape images captured by the dashcam as input. Specifically, the dashcam records video of the scenery while the car is moving, so this video data is input as landscape images. It is assumed that there are multiple buildings in the scenery captured here, and the video recorded by the dashcam includes landscape images of multiple buildings taken from multiple locations.

[0024] The building location-related information receiving unit 102 receives building location-related information, which is information that can identify the location of buildings present in the scenery captured by the drive recorder. Here, GPS information recorded by the drive recorder is received as building location-related information.

[0025] The building image extraction unit 103 generates an image with the building extracted from the landscape image received by the landscape image reception unit 101. Specifically, first, the landscape image is converted into an SfM point cloud using COLMAP, software that generates a 3D point cloud using the SfM algorithm from multiple image data, and distortion correction is performed. Then, the rectangular region containing the building is detected in the landscape image using GroundingDINO, a deep learning model that can perform object detection from image data using text input. Next, the rectangular region containing the building detected by GroundingDINO is segmented using SAM2, a deep learning model that segments the image region, thereby generating an image with the building extracted. Additionally, as an additional processing image, GroundingDINO is used to detect and remove objects that obstruct the building, such as bushes, cars, and streetlights, and an image with image data interpolation is generated using LaMa, a deep learning model that interpolates missing parts from the image. Through this process, for example, from the distortion-corrected landscape image f shown in Figure 4(a), image data b1, b2, b3, b4, and b5 are generated, which are extracted images of the building portion as shown in Figure 4(b). In this way, if the landscape image contains multiple buildings, image data is extracted according to the number of buildings. This generation of image data is performed for all landscape images.

[0026] The building position calculation unit 104 calculates the position of the building as it appears in the building image generated by the building image extraction unit 103. Specifically, it first calculates the shooting position of each image data that makes up the video data from the GPS information received by the building position-related information receiving unit 102 and the video data from the drive recorder received by the landscape image receiving unit 101. Although the reception period of GPS data does not fundamentally match the frame rate of the drive recorder video, the method for matching this is publicly known, so the explanation is omitted. Then, based on this calculation result, the building image extraction unit 103 associates each image data with its shooting position. Next, camera position estimation and camera pose estimation are performed using the COLMAP function, and the camera pose at the shooting position of each image data is associated. After that, for arbitrary feature points such as building corners recognized by GroundingDINO from the SfM point cloud generated by COLMAP, the 3D position is calculated using multiple camera positions and angles.

[0027] The image data receiving unit 105 receives the image data set generated by the building image extraction unit 103. This image data set consists of image data of two or more buildings, each photographed from multiple locations, as a result of the above operation.

[0028] The feature vector acquisition unit 106 acquires feature vectors for each image data by using a visual language model on the image data received by the image data reception unit 105. Here, DINOv2 is used as the visual language model. Feature vectors are arrays created by converting image features such as color, brightness, shape, color distribution, spatial frequency, area, and length into numerical values. The visual language model has the basic function of generating feature vectors with thousands of features converted for any given image data.

[0029] The clustering unit 107 clusters the image data received by the image data receiving unit 105, based on the feature vectors acquired by the feature vector acquisition unit 106, so that groups of image data estimated to have been taken of the same building are considered as one cluster. Here, image data of the same building is assumed to have been taken when feature vectors that are close together in the feature vector space, and clustering is performed based on the density in the feature vector space. DBSCAN is used as the specific clustering method. DBSCAN is an algorithm that uses radius ε and minimum number of points minPts as parameters. Specifically, if there are more than minPts of points within the radius ε of a certain point, that point is considered a core point, and all points within radius ε from that core point, and if there are other core points among those points, all points within radius ε of that other core point are considered to belong to the same cluster. Starting from a core point, all points within the range where the radii are successively connected form a single cluster. Through this process, the image data group g, which schematically represents multiple buildings as shown in Figure 5(a), is clustered into image data groups g1, g2, g3, g4, and g5, which are presumed to represent the same building, as schematically shown in Figure 5(b).

[0030] The map data receiving unit 108 receives map data for the area that includes the scenery captured by the drive recorder, which has been received by the scenery image receiving unit 101. Here, OSM data for this area is used as the map data. The OSM data records attribute information such as the footprint and height of buildings included in the scenery image. Figure 6(a) shows an example of OSM. This OSM is assumed to include the area that includes the scenery image taken by the drive recorder shown in Figure 6(b). In Figure 6(b), the footprints F1 and F2 corresponding to buildings B1 and B2 on the left side are displayed, but the footprints of buildings B3, B4, and B5 on the right side are not displayed. Thus, since OSM is created and edited by an unspecified number of participants, there may be buildings that are not registered. Note that the term "footprint" is not used in OSM data. Footprints are represented by a closed state of a way that represents nodes in an ordered order, and this is used as the footprint.

[0031] The building footprint generation unit 109 generates building footprints on the OSM received by the map data reception unit 108, based on the location of each building calculated by the building location calculation unit 104 and the image data group belonging to each cluster clustered by the clustering unit 107, for buildings estimated to be the subject of the image data group belonging to the cluster. The building footprint generation procedure will be described below based on the schematic diagram of the building footprint generation procedure shown in Figure 7. First, a 3D point cloud consisting of multiple feature points fp of the building is generated from the image data group g1 belonging to one cluster by 3D reconstruction. Specifically, an SfM point cloud is generated as a 3D point cloud from the image data group g1 using COLMAP. The image data group g1 used here is an additional processed image obtained by removing objects that occlude the building and interpolating missing parts in the building image extraction unit 103. Next, the building position calculation unit 104 calculates the location of the building's feature points, and the SfM point cloud is positioned on a space represented by longitude and latitude by matching the location of the same or the most similar feature points in the generated SfM point cloud, thus creating a 3D point cloud s1. Note that Figure 7 is a schematic diagram and therefore has about 10 feature points, but an actual 3D point cloud consists of tens to hundreds of feature points. Then, the SfM point cloud is projected onto a horizontal plane, in this case onto the OSM, and a 2D point cloud t1 is placed on the OSM. Finally, after removing noise and outliers from the 2D point cloud t1, the convex hull, which is the smallest convex set containing the 2D point cloud t1, is generated on the OSM, and this is the building footprint BF1. Note that if there are any changes to the image data sets that make up a cluster, the building footprint generation unit 109 generates a building footprint for the changed cluster each time.

[0032] The footprint integration unit 110 determines the degree of overlap between the building footprint generated by the building footprint generation unit 109 and the footprint of an existing building included in the map data received by the map data reception unit 108. If the overlap exceeds a predetermined standard, the unit integrates the overlapping building footprint with the footprint on the map data. Specifically, since the building footprint is generated on OSM, if the building footprint overlaps with a footprint already present on OSM, both will appear overlapping. The degree of overlap is determined by the ratio of the area of ​​the overlapping portion to the total area of ​​the overlapping region (the sum of the areas of each footprint minus the area of ​​the overlapping portion). In the building footprint generation device 1, this index is also used to determine the degree of overlap for other components. In this case, if the overlap rate is, for example, 20% or more, the overlapping footprint is considered identical to the footprint of the building on OSM that it overlaps with, and is integrated into the footprint on OSM. Here, footprint integration means making the building footprint match the footprint on OSM. In the bottom diagram of Figure 7, the footprint F1 on OSM and the generated building footprint BF1 overlap by more than 20%, so the building footprint BF1 will be integrated into the footprint F1 on OSM. The footprint integration unit 110 also groups the clusters to which the image data that formed the basis of the building footprint integrated into the OSM footprint belongs as OSM clusters, and the clusters that do not belong as non-OSM clusters.

[0033] The cluster reorganization unit 111 verifies whether the image data sets of each generated cluster are appropriate and reorganizes the image data sets that make up the cluster. Note that cluster reorganization is performed only on non-OSM clusters. The cluster reorganization unit 111 consists of a cluster integration unit 111a, a single image footprint generation unit 111b, an image data exclusion unit 111c, an image data transfer unit 111d, and an additional cluster generation unit 111e.

[0034] The cluster integration unit 111a determines the degree of overlap between building footprints and integrates the image data sets that generated building footprints whose overlap exceeds a predetermined standard into the same cluster. Since building footprints are generated on OSM, if the image data sets are separated into different clusters despite being taken of the same building, the building footprints generated from the image data sets belonging to each cluster will overlap on OSM. If this overlap is, for example, 50% or more, they are considered footprints for the same building, and the image data sets belonging to each cluster are integrated into a new cluster. A specific example is shown in Figure 8(a). In Figure 8(a), building footprint BFm and building footprint BFn overlap by 50% or more, so the cluster integration unit 111a considers them footprints for the same building and integrates the image data set belonging to the cluster related to building footprint BFn with the image data set belonging to the cluster related to building footprint BFm. When clusters are merged, the building footprint generation unit 109 generates a new footprint BFmn from the merged clusters.

[0035] The single-image footprint generation unit 111b generates a single-image footprint on OSM based on one image data belonging to each cluster, representing the footprint of the buildings displayed in the image. The single-image footprint generation procedure is explained below based on the schematic diagram of the single-image footprint generation procedure shown in Figure 9. The image data used in the generation of the single-image footprint is an additional processed image obtained by removing elements that obscure the building and interpolating missing parts in the building image extraction unit 103. First, image data b11, b12, b13... are extracted from the image data group g1 that constitutes one cluster, and feature points are extracted for each image data b11, b12, b13... Specifically, feature points fp1, fp2, fp3,... corresponding to each image data b11, b12, b13... are extracted from the feature points that constitute the SfM point cloud of the entire landscape obtained by COLMAP from the original landscape image group. This results in 3D point clouds s11, s12, s13, ..., where the feature points fp1, fp2, fp3, ... of each image data b11, b12, b13... are arranged in 3D. In other words, 3D reconstruction is performed from a single image data using a monocular viewpoint. Subsequently, the 3D point clouds s11, s12, s13, ..., consisting of the obtained 3D feature points fp1, fp2, fp3, ... are projected onto a horizontal plane, in this case onto the OSM, to arrange 2D point clouds. Note that the arrangement of 2D point clouds is omitted in Figure 9. Then, similar to the building footprint generation unit 109, after removing outliers and noise from the 2D point clouds, the smallest convex set containing each 2D point cloud, which is the convex hull, is generated on the OSM, and these are designated as single image footprints OF1, OF2, OF3, ... The single-image footprint generation unit 111b generates a single-image footprint for all image data belonging to a non-OSM cluster.

[0036] The image data exclusion unit 111c compares the building footprint obtained for each cluster with the single image footprint obtained from each image data belonging to that cluster and determines the degree of overlap. If there is a single image footprint that does not overlap or whose overlap is below a predetermined standard, for example, an overlap of 20% or less, the image data from which this single image footprint is derived is excluded from the cluster. For example, as shown in Figure 8(b), suppose single image footprints OFa, OFb, OFc, OFd, and OFE are generated from a group of image data belonging to a cluster related to building footprint BFp. In the figure, single image footprint OFa overlaps with building footprint BFp generated from the cluster by 20% or more, but the other single image footprints OFb, OFc, OFd, and OFE do not overlap with building footprint BFp. In this case, the image data exclusion unit 111c excludes the image data from which single image footprints OFb, OFc, OFd, and OFE are derived from the cluster.

[0037] The image data transfer unit 111d determines the degree of overlap between the image data excluded by the image data exclusion unit 111c and building footprints generated from image data groups belonging to clusters other than the excluded cluster. If a building footprint is found that meets a predetermined criterion, for example, an overlap of 50% or more, the unit transfers the image data that was the source of the single image footprint to the cluster to which the image data group that was the source of this overlapping building footprint belongs. In the case of Figure 8(b), the single image footprint OFb overlaps by 50% or more with a different building footprint BFq, which is different from the building footprint BFp generated from the original cluster. In this case, the image data transfer unit 111d transfers the image data that was the source of the single image footprint OFb to the cluster related to building footprint BFq.

[0038] The additional cluster generation unit 111e determines the degree of overlap between two or more image data that have been excluded by the image data exclusion unit 111c and that could not be transferred by the image data transfer unit 111d. If the determined groups of image data have an overlap of 50% or more, a predetermined criterion is met, and a new cluster is generated that includes the overlapping image data. In the case of Figure 8(b), the single image footprint OFc and the single image footprint OFd overlap by 50% or more. In this case, the additional cluster generation unit 111e generates a new cluster that includes both image data from the image data that formed the basis of the single image footprint OFc and the image data that formed the basis of the single image footprint OFd. Note that in Figure 8(b), OFe does not overlap with any building footprint and does not overlap with any other single image footprint, so it will ultimately be deleted as it does not belong to any cluster.

[0039] When the cluster configuration changes due to the image data exclusion unit 111c, image data transfer unit 111d, and additional cluster generation unit 111e, the building footprint generation unit 109 generates a new building footprint for the cluster whose configuration has changed each time, so the shape of the building footprint also changes each time. For this reason, the image data exclusion unit 111c, image data transfer unit 111d, and additional cluster generation unit 111e repeat the process until there is no more data to exclude from the cluster, or until there is no more image data transfer or additional cluster generation. The image data exclusion unit 111c, image data transfer unit 111d, and additional cluster generation unit 111e determine whether image data belongs to a cluster from a perspective different from that of feature vectors by comparing the building footprint obtained based on the feature vectors of the image data with a single image footprint obtained from individual image data, and can complement the clustering by the clustering unit 107 that uses visual information, thereby contributing to improving the accuracy of clustering.

[0040] The footprint centroid extraction unit 112 extracts the centroid point of the final obtained building footprint as the footprint centroid point. Here, only the footprint centroid points of building footprints related to non-OSM clusters are extracted. The extracted centroid points are used when positioning the three-dimensional models of the buildings that form the footprint.

[0041] The representative image extraction unit 113 extracts representative image data for each cluster obtained after the cluster reorganization unit 111 has finished processing. The representative image data is extracted by calculating the average vector of all feature vectors in the image data group belonging to the cluster, and selecting the image data that generates the feature vector closest to this result as the representative image data. The extracted representative image data is used when generating a three-dimensional model of the buildings depicted in the image data group belonging to the cluster.

[0042] Next, the operation of the building footprint generation device 1 having the above configuration will be explained. Figures 10 and 11 show flowcharts illustrating the operation of the building footprint generation device 1. First, the flowchart in Figure 10 will be explained. The user inputs the video from the drive recorder of the area for which a footprint is to be generated, the GPS data recorded by the drive recorder, and the OSM data of the area in question. As a result, the landscape image receiving unit 101 receives the video from the drive recorder, the building location-related information receiving unit 102 receives the GPS data, and the map data receiving unit 108 receives the OSM data (s101).

[0043] After receiving various data, the building image extraction unit 103 performs distortion correction and removal of obstructions from each image data that makes up the drive recorder video, then extracts building images to generate a group of image data showing buildings, and the building position calculation unit 104 calculates the position of each building (s102). Subsequently, the image data receiving unit 105 receives the generated group of image data (s103).

[0044] Once the image data set has been received, the feature vector acquisition unit 106 acquires a feature vector for each image data using a visual language model (s104). Subsequently, the clustering unit 107 clusters the image data set based on the feature vectors (s105).

[0045] Once the cluster rig is complete, the building footprint generation unit 109 generates a temporary building footprint on the OSM from the image data for each cluster (s106). Next, the footprint integration unit 110 compares the overlap between the generated building footprint and the footprint on the OSM and determines whether the building footprint overlaps with the footprint on the OSM by a certain percentage or more (s107). If the building footprint overlaps with the footprint on the OSM by a certain percentage or more, the building footprint is considered identical to an existing footprint on the OSM and is integrated as a footprint on the OSM (s108). On the other hand, if the overlap with the OSM is less than a certain percentage, the cluster related to that building footprint is classified as a non-OSM cluster (s109). The footprint integration unit 110 performs the above processing from s107 to s109 for all building footprints (s110).

[0046] Next, we will move to the flowchart shown in Figure 11 and mainly explain the operation of the cluster reorganization unit 111. First, the cluster integration unit 111a determines whether there are other building footprints that overlap with the building footprints generated from the non-OSM cluster by a certain percentage or more (s111). If there are other overlapping building footprints, the image data sets constituting the clusters for each building footprint are integrated into a single cluster (s112). After that, the building footprint generation unit 109 generates a new building footprint based on the image data sets constituting the integrated cluster (s113). The cluster integration unit 111a repeats the above process from s111 to s113 for all building footprints (s114).

[0047] Next, the single-image footprint generation unit 111b generates single-image footprints of buildings displayed in the images for all image data belonging to non-OSM clusters (s115). Then, the image data exclusion unit 111c determines whether the overlap between each generated single-image footprint and the building footprint related to the cluster to which the original image data belongs is below a certain percentage (s116). If the overlap between the single-image footprint and the building footprint is below a certain percentage, the image data exclusion unit 111c determines that the image data from which the single-image footprint was generated does not belong to the cluster related to the determined building footprint and excludes this image data from that cluster (s117).

[0048] Furthermore, the image data transfer unit 111d determines whether the overlap between the excluded image data footprint and other building footprints is greater than or equal to a predetermined threshold ratio (s118). If the overlap between the excluded image data footprint and other building footprints is greater than or equal to a predetermined threshold ratio, the image data transfer unit 111d transfers the image data that was the source of this excluded image data footprint to the cluster related to the other building footprint (s119). Subsequently, the building footprint generation unit 109 regenerates building footprints for the clusters from which the image data was excluded and the clusters to which the image data was transferred (s120). The image data exclusion unit 111c and the image data transfer unit 111d perform the above processing from s116 to s120 for all excluded image footprints (s121). Through the above operations, an excluded image footprint that does not overlap with any building footprint is extracted, and as a result, the image data that was the source of this excluded image footprint is extracted.

[0049] Subsequently, the additional cluster generation unit 111e determines whether the single image footprints related to the excluded image data overlap by a certain percentage or more (s122). If the excluded image data overlaps by a certain percentage or more, a new cluster is generated that includes the image data that formed the basis of each overlapping single image footprint (s123). Once a new cluster is generated, the building footprint generation unit 109 generates a building footprint for the new cluster (s123). If the building footprint changes, the process from s116 is repeated. The process from s122 is repeated for all excluded image data (s125). In s125, if there is still excluded image data after processing for all excluded image data is complete, this excluded image data is deleted as it does not belong to any cluster, since it is the image data that formed the basis of a single image footprint that does not overlap with any building footprint or any other single image footprint (s126). At this point, the final building footprint is determined.

[0050] Once the building footprint is determined, the footprint centroid extraction unit 112 calculates and extracts the centroid of the footprint (s127), and the representative image extraction unit 113 extracts representative image data for each cluster (s128).

[0051] As described above, the building footprint generation device 1 can generate building footprints that are not included in OSM from building image data captured by a dashcam. Furthermore, by clustering the building image data based on feature vectors obtained using a visual language model, it can perform highly accurate clustering without requiring additional annotation through zero-shot learning, and generate highly accurate footprints based on this. Additionally, by evaluating the degree of overlap between footprints, the clustering of building image data is made more accurate by evaluating it using a geometric approach in addition to visual features.

[0052] [Embodiment 2] Next, the building footprint generation device 1A according to Embodiment 2 will be described. Figure 12 shows a schematic functional block diagram illustrating the functions of the building footprint generation device 1A formed by incorporating a predetermined program into computer A. The building footprint generation device 1A also functions as a building image clustering device 10A. The difference between the building footprint generation device 1A and the building footprint generation device 1 according to Embodiment 1 is that, instead of the footprint integration unit 110, it has a reference image data group acquisition unit 114 and a clustering adjustment unit 115, and consequently the operation of the clustering unit 107a and the building footprint generation unit 109a changes slightly. The other configurations are the same as those of the building footprint generation device 1, so their explanation will be omitted. The building footprint generation device 1A improves the accuracy of clustering by changing the parameters used for clustering according to the video input from the drive recorder.

[0053] The reference image data acquisition unit 114 acquires a set of image data from the image data reception unit 105 that is deemed to have captured the same building, and uses this set as the reference image data. Here, the map data reception unit 108 extracts a set of image data that is presumed to have captured a building on the OSM data it has received, and uses this set as the reference image data. The specific method for extracting the reference image data is as follows: First, feature points of buildings corresponding to each image data are extracted from the SfM point cloud of the entire landscape obtained by COLMAP based on the original landscape image data, and after projecting them onto the OSM, the centroid of the projected points is obtained as the building position for each image data. Next, if there is an OSM footprint that intersects the line segment connecting the shooting position of the image data and the building position for the image data, or if there is an OSM footprint that exists within a certain range from the building position, for example, within a radius of 40m, then the image data is presumed to have captured the building that constitutes that footprint. If there are multiple OSM footprints that satisfy the conditions, the one closest to the shooting position is selected. By performing this process on each image data, it is possible to extract a set of image data that is presumed to have captured a building on the OSM data, and these become the reference image data set.

[0054] Furthermore, the reference image data acquisition unit 114 groups image data that does not belong to the reference image data group as non-OSM image data groups, and the clustering unit 107a performs clustering only on the non-OSM image data groups. As a result, all clusters generated by clustering by the clustering unit 107a become non-OSM clusters. In addition, the building footprint generation unit 109a generates building footprints only for non-OSM clusters.

[0055] The clustering adjustment unit 115 adjusts the clustering unit 107a so that when it clusters the image data set, there are clusters that match or approximate the reference image data set. As mentioned above, the clustering unit 107a performs clustering using DBSCAN, but DBSCAN uses two parameters, radius ε and minimum points minPts, so the clustering result will differ depending on the values ​​of these parameters. Therefore, the clustering adjustment unit 115 optimizes the values ​​of these parameters so that clusters that match or approximate the reference image data set are generated. Specifically, it uses software that performs Bayesian optimization, a black-box function optimization method, to define an objective function that takes the DBSCAN parameters as input and outputs the clustering evaluation index ARI, and then optimizes the parameters. ARI measures the degree of agreement between the reference image data set and the clustering result closest to it. The parameters determined by the clustering adjustment unit 115 will be used for clustering in the clustering unit 107a.

[0056] Next, the operation of the building footprint generation device 1A will be described. Figure 13 shows a flowchart representing the first half of the operation of the building footprint generation device 1A. This flowchart corresponds to the flowchart shown in Figure 10 for the building footprint generation device 1. Regarding the flowchart shown in Figure 11, which mainly represents the operation of the cluster reorganization unit 111 and the second half of the operation of the building footprint generation device 1, the building footprint generation device 1A operates similarly, so the explanation of the operation from the flowchart shown in Figure 13 onwards will be omitted.

[0057] First, steps s201 to s204 are the same as steps s101 to s104 in the flowchart shown in Figure 10 for the building footprint generation device 1. Through these steps, the building footprint generation device 1A receives the building image data and OSM data, calculates the building position, and obtains feature vectors for each image data. Next, the reference image data acquisition unit 114 extracts the building footprint and height information from the OSM data and generates a virtual 3D model at the footprint location. Then, using ray tracing, it generates a ray-traced image that simulates how the virtual 3D model generated from the image data's shooting position and camera pose would appear, and acquires feature vectors from the ray-traced image using a visual language model via the feature vector acquisition unit 106. The feature vectors of the image data and the feature vectors of the ray-traced image are then compared, and it is determined whether the two are approximate based on whether the distance in the vector space is less than or equal to a predetermined value (s205). If there are multiple virtual 3D models, the ray-traced image and image data are compared for each 3D model. If it is determined that they are approximate, the image data subject to the determination is classified as reference image data related to the OSM footprint that was the basis for the virtual 3D model used to simulate the approximate ray-traced image (s206). On the other hand, if there is no approximate ray-traced image, the image data subject to the determination is classified as a non-OSM image (s207). The operations described in s205 to s207 are repeated until all image data processing is complete (s208).

[0058] Once the reference image data set is obtained, the clustering adjustment unit 115 uses the reference image data set to determine the DBSCAN parameters to be used by the clustering unit 107a using Bayesian optimization (s209). Once the DBSCAN parameters are determined, the clustering unit 107a uses these parameters to perform clustering only on non-OSM image data and generate non-OSM clusters (s210). Then, the building footprint generation unit 109a generates building footprints on OSM for only the non-OSM clusters (s211). From this point onward, the building footprint generation device 1A performs the same operations as the building footprint generation device 1 shown in the flowchart of Figure 11.

[0059] Through the operations described above, the building footprint generation device 1A extracts a set of reference image data based on the footprints in OSM and adjusts the parameters of the clustering unit 107 based on this. Since the reference image data set is captured by the drive recorder under the same shooting conditions along with other image data sets, adjusting the clustering unit 107 using the reference image data set allows the clustering to be flexibly adapted to individual shooting conditions, thereby making the clustering of building image data by the clustering unit 107 more accurate.

[0060] [Differentiation] (1) In the above embodiment, the AI ​​models and software are linked with the same computer A as the building footprint generation devices 1 and 1A. However, these AI models may be linked with those installed on an external computer connected via a communication network, or some or all of them may be incorporated into the building footprint generation devices 1 and 1A. Furthermore, the AI ​​models and software used are examples only, and any AI models that can perform similar functions can be adopted.

[0061] (2) In the above embodiment, video from a dashcam is used as image data of buildings, but various types of image data can be used if a set of image data is obtained from taking pictures of multiple buildings from two or more positions. For example, various types of image data can be used, such as video data taken by a drone, image data taken by a person, video from a dashcam, and mixed image data. Furthermore, with regard to dashcam video, it is possible to process only a portion of the images that meet specific conditions, rather than processing all of the landscape images. For example, in order to improve the efficiency and accuracy of processing, images in which the proportion of buildings in the image or the number and density of building feature points do not meet a predetermined standard (threshold) may be excluded from processing.

[0062] (3) In the above embodiment, GPS information from a drive recorder is used as building location-related information, but various types of information can be used as long as they can be used to calculate the location of the photographed building. For example, if a 2D point cloud obtained by projecting an SfM point cloud of a landscape image onto a horizontal plane is adjusted in size and position and placed on top of an OSM footmap, and the approximate location of a building not on the OSM can be calculated from its relative position to an OSM building whose location information is known, then the OSM map can be used as building-related information. In addition, various types of data can be used, such as data that associates image data of a building photographed by a person with the location information of that building.

[0063] (4) In the above embodiment, the clustering units 107 and 107a cluster the image data set using DBSCAN based on feature vectors. However, it is possible to use any method that performs clustering based on the proximity of feature vectors, such as using OPTICS, an algorithm that uses the same density as a criterion, or using a method that uses the distance between vectors as a criterion. In this case, the parameters will change, so the clustering adjustment unit 115 will determine the parameters to be used in the method. Alternatively, instead of clustering by algorithm, the image data set may be clustered using a cluster estimation model that has been trained to classify image data sets of the same building as the same cluster, taking feature vectors as input.

[0064] (5) In the above embodiment, a footprint is generated from a 2D point cloud projected onto a horizontal plane using a convex hull, but the method is not limited to using a convex hull. For example, in situations where it is desirable to approximate with a simpler shape, such as when the shape of a building can be considered to be roughly rectangular, the smallest rotating circumscribed rectangle, which is the rectangle with the smallest area encompassing the point cloud, can also be used. In addition, other known methods for forming contours from 2D point clouds, such as alpha shapes, can be appropriately selected. Furthermore, these can be combined, or new methods can be adopted.

[0065] (6) In the above embodiment, the degree of overlap between building footprints and footprints on OSM is evaluated by the overlap ratio, but any evaluation method can be used as long as the degree of overlap can be determined, and the criteria value for the judgment can also be set arbitrarily.

[0066] (7) In Embodiment 1 described above, when integrating the building footprint with the footprint on OSM, the OSM footprint is used as the reference for integration. However, the building footprint may also be used for integration, or the shapes of both the building footprint and the OSM footprint may be combined based on the midpoint between their centroids. For example, if the building footprint generated from a large number of image data is more detailed and accurate than the simplified shape of OSM, the highly accurate shape can be used as the reference, and the OSM data can be used as an aid for alignment. On the other hand, if both sets of data have advantages and disadvantages and can be considered to have similar reliability, instead of prioritizing one over the other, the shape information of both can be merged to generate an intermediate shape. This cancels out positional deviations and shape errors between them, making it possible to obtain a more stable and accurate footprint. In this way, by adapting the optimal integration method according to the quality of the data, the accuracy of the footprint can be further improved.

[0067] (8) In the above embodiment 1, the footprint integration unit 110 processes all footprints on OSM, but a function may be provided to exclude them from processing at the user's request. In addition, with regard to building footprints, if the user can identify a specific building in advance and does not want it processed, a function may be provided to exclude it from processing at the user's request.

[0068] (9) In Embodiment 2 described above, reference image data sets are acquired for all footprints on OSM. However, for the purpose of selecting parameters in the clustering adjustment unit 115, it is sufficient to acquire reference image data sets for at least one footprint. In this case, the extraction of non-OSM clusters can be done in the same way as in Embodiment 1, by generating building footprints for all clusters using the building footprint generation unit 109 and then selecting non-OSM clusters using the footprint integration unit 110.

[0069] (10) In the above embodiment, the feature vector acquisition unit 106 uses image data before removing objects that obstruct the building, and the building footprint generation unit 109 and the single-image footprint generation unit 111b use additionally processed images that have been interpolated by removing objects that obstruct the building. However, the feature vector acquisition unit 106 can also use additionally processed images or use additionally processed images in combination. Furthermore, images that have undergone other additional processing may also be used.

[0070] (11) Although the above embodiment shows that processing is performed using only image data from one drive recorder, it is also possible to store feature vectors, reference image data, etc., in a database each time image data from a drive recorder is processed, and to perform clustering, parameter adjustment, etc., using the stored data. This makes it possible to expect successive improvements in accuracy with use. Furthermore, timestamps may be added when saving image data and feature vectors to the database. This allows the saved data to be used as a long-term index for detecting building renovations, updates, and age-related changes, or as auxiliary information when adjusting and improving large-scale building recognition models.

[0071] [summary] (1) The building image clustering device according to the present invention, which has as a function the building footprint generation device 1, 1A according to the present invention, includes an image data receiving unit 105 that receives input of image data groups in which two or more buildings are each photographed from multiple locations, a feature vector acquisition unit 106 that acquires a feature vector for each image data by using a visual language model on the image data group, and a clustering unit 107 that clusters the image data group as a cluster based on the feature vector, with the image data group estimated to have been photographed of the same building being treated as one cluster. With this configuration, the building image clustering device according to the present invention can cluster groups of image data of the same building from a group of image data of two or more buildings taken from multiple locations, based on feature vectors obtained from a visual language model. The feature vectors obtained from the visual language model are versatile because they can extract visual features from any image data through zero-shot learning, and can flexibly handle clustering even if the shooting conditions of each building image differ.

[0072] (2) The building image clustering device according to the present invention has a landscape image receiving unit 101 that receives landscape images including the building taken from multiple locations, and a building image extraction unit 103 that generates an image of the building extracted from the landscape image, and the image data receiving unit 105 receives the multiple image data generated by the building image extraction unit as the image data group. This configuration allows for the creation of a set of image data of buildings using broad images of the surrounding landscape, thus reducing the effort required compared to individually photographing each building.

[0073] (3) The building image clustering device according to the present invention includes a reference image data acquisition unit 114 that acquires a group of image data from the group of image data that is deemed to have been taken of the same building as a reference image data group, and a clustering adjustment unit 115 that adjusts the clustering unit 107a so that when clustering is performed by the clustering unit 107a there exists a cluster that matches or approximates the reference image data group. With this configuration, the clustering unit adjusts based on the reference data set and the corresponding image data set. In particular, when the image data sets are captured under the same or similar conditions, the clustering unit adjusts to match the same or similar shooting conditions, thereby improving the accuracy of clustering by the clustering unit.

[0074] (4) The building image clustering device according to the present invention includes: a building location-related information receiving unit 102 that receives building location-related information which is information that can identify the location of a building displayed in each image of the image data group; a building footprint generation unit 109 that generates a building footprint as a building footprint based on the building location-related information and the image data group belonging to the clusters clustered by the clustering units 107, 107a, and projects a 2D point cloud obtained by 3D reconstruction of a building estimated to be the target of photography based on the image data group onto a horizontal plane; and a cluster integration unit 111a that determines the degree of overlap between the building footprints and integrates the image data groups that were used to generate building footprints whose overlap is greater than a predetermined standard into the same cluster. With this configuration, even if the clustering units 107 and 107a divide the image data of the same building into two or more clusters, the clusters that have been separated are integrated using a geometric method based on the overlap of footprints, which is different from clustering based on visual information using feature vectors, thus making the clustering more optimal.

[0075] (5) The building image clustering device according to the present invention includes a building location-related information receiving unit 102 which receives building location-related information which is information that can identify the location of a building displayed in each image of the image data group, and building footprint generation units 109, 109a which generate a building footprint as a building footprint based on the building location-related information and the image data group belonging to the cluster clustered by the clustering unit, and generate a two-dimensional point cloud obtained by projecting a three-dimensional point cloud obtained by three-dimensional reconstruction of the building estimated to be the target of photography based on the image data group onto a horizontal plane, and the building location-related information and the cluster The system includes a single-image footprint generation unit 111b that generates a single-image footprint of a building based on a 2D point cloud obtained by projecting a 3D point cloud, which is obtained by 3D reconstruction of the building displayed in the image from a monocular viewpoint, onto a horizontal plane, based on each image data belonging to each raster, and an image data exclusion unit 111c that determines the degree of overlap between the single-image footprint belonging to each cluster and the building footprint obtained from the image data group belonging to that cluster, and excludes from the cluster any image data that does not overlap or whose overlap is below a predetermined standard the image data that was used to generate the single-image footprint. With this configuration, if the image data clustered by the clustering units 107 and 107a contains image data that does not depict the assumed building, it is possible to extract and exclude the image data that does not depict the assumed building using a geometric method based on the overlap of footprints, which is different from clustering based on visual information using feature vectors.

[0076] (6) The building image clustering device according to the present invention has an image data transfer unit 111d that determines whether there is a building footprint whose single image footprint generated from the image data has overlapped with the image data excluded by the image data exclusion unit 111c by a predetermined standard or more, and if there is, transfers the excluded image data to the cluster to which the group of image data that generated the building footprint belongs. With this configuration, if the image data excluded by the image data exclusion unit 111c is image data that should belong to another cluster, the excluded image data can be transferred to the cluster to which it originally belonged by a geometric method of overlapping footprints, which is different from clustering based on visual information using feature vectors.

[0077] (7) The building image clustering device according to the present invention has an additional cluster generation unit 111e which generates a new cluster of two or more image data groups that have been excluded by the image data exclusion unit 111c, which cannot be transferred by the image data transfer unit 111d, and whose image footprints overlap by a predetermined standard or more. With this configuration, if two or more sets of image data excluded by the image data exclusion unit 111c are sets of image data of buildings that are not the target of any of the clusters, new clusters can be generated from these excluded image data using a geometric method of footprint overlap, which is different from clustering based on visual information using feature vectors.

[0078] (8) The building image clustering device according to the present invention has a representative image extraction unit 113 that extracts image data that generates a feature vector closest to the average vector of all feature vectors of the image data group constituting the final cluster as representative image data of the cluster. With this configuration, the representative image data can be estimated to be the image data that best represents the characteristics of the photographed building. Therefore, when generating a footprint from a group of image data belonging to a cluster and then placing a 3D model at the location of the footprint, the image data extracted by the representative image extraction unit 113 can be used to obtain the image data that best represents the characteristics of the target building.

[0079] (9) The building footprint generation devices 1 and 1A according to the present invention include a building image clustering device according to any of (1) to (6), a building location-related information receiving unit 102 that receives building location-related information which is information that can identify the location of a building displayed in each image of the image data group, and building footprint generation units 109 and 109a that generate the building footprint based on the building location-related information and the image data group belonging to each cluster clustered by the building clustering device. With this configuration, the building footprint generation devices 1 and 1A can obtain the footprint of each building from a set of image data of two or more buildings taken from multiple locations, based on feature vectors obtained from a visual language model. The feature vectors obtained from the visual language model are versatile because they can extract visual features from any image data through zero-shot learning, and can flexibly generate footprints for each building even if the shooting conditions of each building image differ.

[0080] (10) The building footprint generation device 1 according to the present invention includes a map data receiving unit 108 that receives map data including a known building footprint which is the footprint of a known building that encompasses the area in which the building is located, and a footprint integrating unit 110 that determines whether there are any building footprints generated by the building footprint generation unit 109 that overlap with the known building footprint of the map data by a predetermined standard or more, and if there are, integrates the building footprint that is determined to exist with the known building footprint that overlaps it. With this configuration, if there are buildings whose footprints already exist on the existing map data, the footprints of those buildings can be integrated with the building footprints generated by the building footprint generation unit 109, thereby effectively utilizing the existing building footprints. [Explanation of Symbols]

[0081] 1. 1A Building footprint generation device 10, 10A Building Image Clustering Device 101 Landscape Image Reception Department 102 Building Location-Related Information Reception Department 103 Building image cropping section 104 Building Location Calculation Unit 105 Image Data Reception Department 106 Feature Vector Acquisition Unit 107 Clustering section 108 Map Data Reception Department 109 Building Footprint Generation Unit 110 Footprint Integration Unit 111 Cluster Reorganization Department 111a Cluster Integration Unit 111b Image footprint generation unit 111c Image data exclusion section 111d Image Data Transfer Unit 111e Additional cluster generation unit 112 Footprint centroid extraction unit 113 Representative Image Extraction Unit 114 Reference Image Data Acquisition Unit 115 Clustering adjustment unit

Claims

1. An image data receiving unit that accepts a group of image data taken from multiple locations of two or more buildings, A feature vector acquisition unit that acquires a feature vector for each image data by using a visual language model on the aforementioned image data set, A clustering unit that clusters image data groups that are estimated to have been taken of the same building based on the feature vectors, A reference image data acquisition unit acquires a set of image data from the aforementioned set of image data that is deemed to have photographed the same building, as a reference image data set. A clustering adjustment unit adjusts the clustering unit so that when clustered by the clustering unit, there are clusters that match or approximate the reference image data group. A building image clustering device having the following features.

2. An image data receiving unit that receives input of a group of image data taken of two or more buildings from multiple locations, A feature vector acquisition unit that acquires a feature vector for each image data by using a visual language model on the aforementioned image data set, A clustering unit that clusters image data groups that are estimated to have been taken of the same building based on the feature vectors, A building location-related information receiving unit receives building location-related information, which is information that can identify the location of a building displayed in each image of the aforementioned image data group. A building footprint generation unit generates the building footprint as a building footprint based on the building location-related information and the image data group belonging to each cluster clustered by the clustering unit. A building image clustering device having the following features.

3. The building image clustering device according to claim 2, wherein the building footprint generation unit generates a building footprint as a building footprint based on a two-dimensional point cloud obtained by projecting a three-dimensional point cloud obtained by three-dimensionally reconstructing a building estimated to be the subject of photography based on the image data group onto a horizontal plane.

4. A cluster integration unit that determines the degree of overlap between the building footprints and integrates the sets of image data that generated the building footprints whose overlap exceeds a predetermined standard into the same cluster. The building image clustering device according to claim 3.

5. A single-image footprint generation unit generates a single-image footprint of a building based on the building location-related information and the image data belonging to each of the clusters, by projecting a 2D point cloud obtained by 3D reconstruction of the building displayed in the image from a monocular viewpoint onto a horizontal plane, thereby generating a single-image footprint of the building. For each cluster, the degree of overlap between the single image footprint belonging to that cluster and the building footprint obtained from the image data group belonging to that cluster is determined, and an image data exclusion unit excludes from the cluster the image data that generated the single image footprint that does not overlap or whose overlap is below a predetermined standard. A building image clustering device according to claim 3, having the following features.

6. The building image clustering device according to claim 5, further comprising an image data transfer unit that, with respect to the image data excluded by the image data exclusion unit, determines whether there exists a building footprint whose single image footprint generated from the image data overlaps with a predetermined standard or more, and if such a building footprint exists, transfers the excluded image data to the cluster to which the group of image data that generated the building footprint belongs.

7. The building image clustering device according to claim 6, further comprising an additional cluster generation unit that generates a new cluster from two or more image data groups excluded by the image data exclusion unit, which cannot be transferred by the image data transfer unit, and whose image footprints overlap by a predetermined standard or more.

8. The building image clustering apparatus according to claim 1, further comprising a representative image extraction unit that extracts image data that generates a feature vector closest to the average vector of all feature vectors of the image data group constituting the final obtained cluster as representative image data of the cluster.

Images