Feature amount space generation device, feature amount space generation method, and program

Abstract

A feature amount space generation device according to one embodiment of the present invention comprises an acquisition unit, a generation unit, an extraction unit, and a processing unit. The acquisition unit acquires a plurality of 2D images obtained by capturing a predetermined space, 3D point group data obtained by three-dimensionally measuring the predetermined space, and parameter information including information on imaging position and imaging orientation corresponding to each of the plurality of 2D images. The generation unit executes projective transformation of 3D point group data based on imaging position and imaging orientation in a predetermined space for each of the plurality of 2D images, generates single local feature amount point group data related to a local feature amount of a specific point group from viewpoints corresponding to each of the plurality of 2D images, and generates integrated local feature amount point group data by integrating all of the single local feature amount point group data excluding object point group data of an arbitrary object 2D image. The extraction unit performs difference detection between the integrated local feature amount point group data and the object point group data, and extracts, as feature amount missing voxel data, a set of voxels including a point group having a local feature amount that exists in the object point group data and does not exist in the integrated local feature amount point group data. The processing unit sorts the object 2D image into either a first list, used for generating a feature amount space of the predetermined space, or a second list, not used for generating the feature amount space, on the basis of the feature amount missing voxel data.

Description

Feature Space Generation Device, Feature Space Generation Method, and Program

[0007] ,

[0001] The present invention relates to a feature space generation device, a feature space generation method, and a program.

[0002] Conventionally, there is a technique for calculating the three-dimensional positions of image feature points extracted from an image of a predetermined space and generating a feature space. By utilizing this feature space, it becomes possible to estimate the shooting position and orientation of a camera image captured by a camera in a predetermined space.

[0003] As a method for calculating the three-dimensional positions of the above-mentioned image feature points, SfM (Structure from Motion) is well known. It is known that in SfM, the higher the accuracy of three-dimensional restoration, the more images are input.

[0004] On the other hand, preparing a large number of images to improve the three-dimensional restoration accuracy takes a long time for shooting. Therefore, a method has been proposed for calculating the three-dimensional positions of image feature points by combining a small number of images and LiDAR (Light Detection And Ranging) point clouds (see Patent Document 1).

[0005] In the proposed method that combines an image and a LiDAR point cloud (hereinafter referred to as the hybrid method), by overlapping a point cloud image obtained by projective transformation of the LiDAR point cloud with one image, all the feature points of one image can be three-dimensionalized, and a feature space can be generated. Therefore, the hybrid method has the advantage that the number of necessary images is small compared to a method of generating a feature space using only images such as SfM.

[0006] International Application No. PCT / JP2023 / 019974 Specification

[0007] However, when capturing and measuring the images and LiDAR point clouds required for the hybrid method, the minimum number of images needed is unknown, or more images and measurements may be taken than necessary to avoid revisiting, re-capturing, and re-measuring a given space. In the hybrid method, the LiDAR point cloud is projected onto all input images and incorporated into the feature space, but a high-density LiDAR point cloud is required to associate image feature points with the LiDAR point cloud. A high-density LiDAR point cloud has the characteristic that the amount of point cloud data contained in the data itself is large, and the processing time required for association increases as the amount of point cloud data increases. In addition, information from images that are not originally necessary is incorporated into the feature space, which increases the capacity of the feature space.

[0008] Therefore, when more captured images than necessary are input in the hybrid system, there are two problems: (1) the processing time for LiDAR point clouds becomes longer, and (2) the capacity of the feature space becomes larger.

[0009] While it is possible to manually remove unnecessary images after shooting, it is difficult for a person to visually determine the minimum necessary images. Furthermore, simply removing images may lead to the removal of essential images, potentially degrading the accuracy of estimating the camera's shooting position and orientation using the feature space.

[0010] A feature space generation device according to one aspect of the present invention comprises an acquisition unit, a generation unit, an extraction unit, and a processing unit. The acquisition unit acquires a plurality of 2D images taken of a predetermined space, 3D point cloud data obtained by three-dimensional measurement of the predetermined space, and parameter information including information on the shooting position and shooting orientation corresponding to each of the plurality of 2D images. The generation unit performs a projection transformation of the 3D point cloud data based on the shooting position and shooting orientation in the predetermined space for each of the plurality of 2D images, generates single local feature point cloud data relating to the local features of the point cloud identified from the viewpoint corresponding to each of the plurality of 2D images, and generates integrated local feature point cloud data by integrating all single local feature point cloud data excluding the target point cloud data of an arbitrary target 2D image. The extraction unit detects the difference between the integrated local feature point cloud data and the target point cloud data, and extracts a set of voxels containing point clouds with local features that exist in the target point cloud data but not in the integrated local feature point cloud data as feature-deficient voxel data. The processing unit sorts the target 2D image into either a first list used for generating a feature space for a predetermined space or a second list not used for generating a feature space, based on the feature-deficient voxel data.

[0011] According to the present invention, when generating a feature space, unnecessary images can be automatically excluded from images that have been captured in excess of what is needed.

[0012] Figure 1 is a block diagram showing an example of the overview of processing by the feature space generation device according to the first embodiment. Figure 2 is a block diagram showing an example of the hardware configuration of the feature space generation device according to the first embodiment. Figure 3 is a block diagram showing an example of the functional configuration of the feature space generation device according to the first embodiment. Figure 4 is a flowchart showing an example of list output processing in the feature space generation device according to the first embodiment. Figure 5 is a flowchart showing a specific example of exclusion processing based on 2D images in the flowchart of Figure 4. Figure 6 is a diagram illustrating the relationship between the map, shooting location, and partition group in the first embodiment. Figure 7 is a flowchart showing a specific example of sorting processing in the flowchart of Figure 6. Figure 8 is a diagram illustrating multiple shooting locations included in a specific partition among the partition group in Figure 6. Figure 9 is a diagram illustrating multiple camera images corresponding to the multiple shooting locations in Figure 8. Figure 10 is a block diagram showing an example of the functional configuration of the feature space generation device according to the second embodiment. Figure 11 is a flowchart showing an example of list output processing in the feature space generation device according to the second embodiment. Figure 12 is a flowchart showing a specific example of exclusion removal processing in the flowchart of Figure 11. Figure 13 is a diagram illustrating an example of spatial loss in the second embodiment. Figure 14 is a diagram illustrating an example of the camera image being revived and the reduction of spatial loss in the second embodiment. Figure 15 is a block diagram illustrating an example of the functional configuration of the feature space generation device according to the third embodiment. Figure 16 is a flowchart illustrating an example of list output processing in the feature space generation device according to the third embodiment. Figure 17 is a flowchart illustrating a specific example of exclusion processing based on 3D point cloud data in the flowchart of Figure 16. Figure 18 is a block diagram illustrating an example of the functional configuration of the feature space generation device according to the fourth embodiment. Figure 19 is a flowchart illustrating an example of list output processing in the feature space generation device according to the fourth embodiment. Figure 20 is a flowchart illustrating a specific example of exclusion processing based on 3D point cloud data in the flowchart of Figure 19. Figure 21 is a block diagram illustrating an example of the functional configuration of the feature space generation device according to the fifth embodiment.Figure 22 is a flowchart showing an example of list output processing in a feature space generation device according to the fifth embodiment.

[0013] Embodiments of the present invention will be described below with reference to the drawings.

[0014] <First Embodiment> [Configuration] Figure 1 is a block diagram showing an example of the overview of processing by the feature space generation device according to the first embodiment. The feature space generation device 100 in Figure 1 can select the minimum set of images necessary to generate a feature space for a predetermined space when it receives data from multiple 2D images (2D image data) taken of a predetermined space, 3D point cloud data obtained by three-dimensional measurement of the predetermined space, and parameter information. The feature space is information in which feature quantities (feature vectors) are assigned to key points detected from the 3D point cloud data.

[0015] A 2D image is, for example, a camera image taken by an optical camera. The file format for a 2D image is, for example, a format that contains the two-dimensional position of pixels and RGB information (RGB values ​​in the RGB color space). Specifically, 2D image file formats include, for example, "JPEG" and "PNG". Hereafter, a 2D image may simply be referred to as "image".

[0016] 3D point cloud data includes point clouds measured by LiDAR at multiple different measurement locations, such as in LiDAR-SLAM. Each point in the 3D point cloud data includes, for example, positional information in three-dimensional space. The file format of the 3D point cloud data is, for example, a format that contains the three-dimensional positional information of the point cloud. Specifically, the file formats for 3D point cloud data include "E57", "PLY", and "LAS".

[0017] The parameter information includes external parameter information related to the camera that captured the 2D image. This external parameter information includes the camera's position and orientation in three-dimensional space when the 2D image was captured. The orientation information is, for example, information that allows for the calculation of the camera's orientation in three-dimensional space. Specifically, the orientation information includes information such as Euler angles and quaternions. The "camera's position in three-dimensional space" may also be called the "shooting position." The "camera's orientation in three-dimensional space" may also be called the "shooting direction" or "shooting orientation."

[0018] Figure 2 is a block diagram showing an example of the hardware configuration of a feature space generation device according to the first embodiment. The feature space generation device 100 in Figure 2 includes a control circuit 101, storage 102, communication module 103, interface 104, and drive 105.

[0019] The control circuit 101 is a circuit that controls all components of the feature space generation device 100 as a whole. The control circuit 101 includes a CPU (Central Processing Unit), RAM (Random Access Memory), and ROM (Read Only Memory). The ROM of the control circuit 101 stores programs used in various processes in the feature space generation device 100. The CPU of the control circuit 101 controls the entire feature space generation device 100 according to the programs stored in the ROM of the control circuit 101. The RAM of the control circuit 101 is used as a workspace for the CPU of the control circuit 101.

[0020] The storage 102 is composed of, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), or flash memory. Information used in various processes in the feature space generation device 100 is stored in the storage 102.

[0021] The communication module 103 is a circuit used for sending and receiving data with other devices (external devices).

[0022] Interface 104 is an interface primarily used for communication between the user and the control circuit 101. Interface 104 includes, for example, input devices and output devices. Input devices include, for example, a microphone, camera, touch panel, and operation buttons. Output devices include, for example, a speaker and a display.

[0023] Drive 105 is a device for reading software stored on the storage medium 106. Drive 105 includes, for example, a CD (Compact Disk) drive or a DVD (Digital Versatile Disk) drive.

[0024] The storage medium 106 is a medium for storing software by electrical, magnetic, optical, mechanical, or chemical means. The storage medium 106 may also store programs for executing various processes in the feature space generation device 100.

[0025] The storage medium 106 may also be a USB (Universal Serial Bus) memory. If the storage medium 106 is a USB memory, it is connected to, for example, the communication module 103. In this case, the feature space generation device 100 does not need to have a drive 105.

[0026] Figure 3 is a block diagram showing an example of the functional configuration of a feature space generation device according to the first embodiment. The feature space generation device 100 in Figure 3 includes an acquisition unit 110, a classification unit 120, a feature point extraction unit 130, an unclassified list generation unit 140, a first processing unit 150, and an output unit 160.

[0027] To realize the functional configuration shown in Figure 3, the CPU of the control circuit 101 in Figure 2 loads the program stored in the ROM or storage medium 106 of the control circuit 101 into the RAM of the control circuit 101. The CPU of the control circuit 101 then interprets and executes the program loaded into the RAM of the control circuit 101. As a result, the feature space generation device 100 functions as a computer comprising an acquisition unit 110, a classification unit 120, a feature point extraction unit 130, an unclassified list generation unit 140, a first processing unit 150, and an output unit 160.

[0028] The acquisition unit 110 acquires 2D image data, 3D point cloud data, and parameter information for a predetermined space through user operation. For example, the acquisition unit 110 may acquire this data and information from storage 102 or an external device.

[0029] The classification unit 120 classifies the acquired 2D image data into multiple groups. Specifically, the classification unit 120 classifies the acquired 2D image data into multiple groups related to the shooting location or shooting target in a predetermined space.

[0030] A group relating to the shooting position is, for example, a group obtained by dividing a given space in the horizontal plane into multiple sections. The classification unit 120 assigns the 2D images to be classified to the group corresponding to the shooting position of the 2D images to be classified. Note that the "group relating to the shooting position" may also be called a "section group".

[0031] The groups related to the subjects being photographed are, for example, groups classified according to the similarity of the global features of the 2D images to be classified. The classification unit 120 assigns the 2D images to be classified to groups according to the similarity of the global features of the 2D images to be classified. In this case, the feature space generation device 100 is assumed to have extracted the global features of the 2D images in advance. The feature space generation device 100 may also extract local features when extracting global features.

[0032] Global features are features extracted from the entirety of a 2D image. Global features can be extracted using methods such as VLAD (Vector of Locally Aggregated Descriptors). VLAD aggregates multiple local features extracted from a 2D image into a single feature vector.

[0033] Local features are features extracted from a portion of an image. Local features can be extracted using techniques such as SIFT (Scale-Invariant Feature Transform).

[0034] Furthermore, global features may be extracted using NetVLAD or other neural network-based methods instead of VLAD, and the extraction method is not particularly restricted. Similarly, local features may be extracted using SuperPoint or other neural network-based methods instead of SIFT, and the extraction method is not particularly restricted.

[0035] The feature point extraction unit 130 extracts feature points from each 2D image data. Feature points are, for example, edges and corners included in the image. The feature point extraction unit 130 may also extract local features, which are characteristic quantities of the feature points.

[0036] The unclassified list generation unit 140 generates a list of 2D images (unclassified list) for each of the multiple groups. The unclassified list may be arranged in descending order of the number of local feature points. By arranging the 2D images in descending order of the number of local feature points in the unclassified list, images with a large number of local feature points, which are advantageous for position estimation, are more likely to remain.

[0037] The first processing unit 150 sorts the 2D images included in the unclassified list into an accept list and an exclusion list for each of the multiple groups. Specifically, the first processing unit 150 selects a group to be processed from the multiple groups and selects one arbitrary image from the unclassified list of the selected group. The first processing unit 150 sequentially compares the selected image with the other images included in the unclassified list, removes any other images that satisfy the comparison conditions from the unclassified list, and adds them to the exclusion list. The first processing unit 150 removes the selected image, which no longer has any comparison targets, from the unclassified list and adds it to the accept list. The first processing unit 150 repeatedly performs this sorting process until all images included in the unclassified list of the selected group are sorted into the accept list and the exclusion list. Furthermore, from the perspective of making images with a large number of local feature points, which are advantageous for position estimation, more likely to remain, it is desirable for the first processing unit 150 to select the image with the most feature points as the arbitrary image.

[0038] The above comparison conditions are based, for example, on the similarity of global features and the number of matches (matching rate) of local features. Specifically, the first processing unit 150 determines that the comparison conditions are met if the similarity (e.g., cosine similarity) between the global features of the selected image and the global features of other images is equal to or greater than a predetermined threshold (first threshold), and the number of matches between the local features of the selected image and the local features of other images is equal to or greater than a predetermined threshold (second threshold).

[0039] For example, the comparison conditions may also be based on the similarity of the camera lens direction among the shooting postures and the number of matching local features. Specifically, the first processing unit 150 may determine that the comparison conditions are met if the similarity (e.g., cosine similarity) between the camera lens direction of the selected image and the camera lens direction of the other image is greater than or equal to a predetermined threshold (third threshold), and the number of matching local features between the selected image and the local features of the other image is greater than or equal to a predetermined threshold.

[0040] The comparison criteria may be based solely on the number of matching local features. Specifically, the first processing unit 150 may determine that the comparison criteria are met if the number of matching local features between the selected image and the local features of other images is equal to or greater than a predetermined threshold.

[0041] The first processing unit 150 may also extract and calculate global features, local features, and shooting posture related to the 2D image. Specifically, the first processing unit 150 extracts global features from the 2D image using VLAD or NetVLAD as described above. The first processing unit 150 extracts local features from the 2D image using SIFT or SuperPoint as described above. The first processing unit 150 calculates the shooting posture based on the posture information associated with the 2D image.

[0042] The output unit 160 outputs an acceptance list for all 2D images sorted by the first processing unit 150. The acceptance list output by the output unit 160 may be divided by the group to which the 2D images belong, or each list may contain information about the group to which the 2D images belong. This allows for processing of each group if there are subsequent processes.

[0043] The configuration of the feature space generation device according to the first embodiment has been described above. Next, the operation of the feature space generation device according to the first embodiment will be described.

[0044] [Operation] FIG. 4 is a flowchart showing an example of list output processing in the feature space generation device according to the first embodiment. The list output processing is a process of outputting an adoption list used for generating a feature space for a plurality of 2D images obtained by photographing a predetermined space. The flowchart of FIG. 4 starts, for example, when a user executes the list output processing.

[0045] (Step ST110) The acquisition unit 110 acquires a plurality of 2D images, 3D point cloud data, and parameter information.

[0046] (Step ST120) The feature space generation device 100 excludes 2D images other than the processing target based on the plurality of 2D images. Hereinafter, the processing in step ST120 will be referred to as "exclusion processing based on 2D images". Hereinafter, a specific example of the exclusion processing based on 2D images will be described using the flowchart of FIG. 5.

[0047] FIG. 5 is a flowchart showing a specific example of the exclusion processing based on 2D images in the flowchart of FIG. 4. The flowchart of FIG. 5 transitions from step ST110 of FIG. 4 and starts from step ST201.

[0048] (Step ST201) The classification unit 120 classifies all the 2D images into a plurality of groups. Hereinafter, a specific example of classifying 2D images into partition groups will be described using FIG. 6.

[0049] Figure 6 illustrates the relationship between the map, shooting locations, and section groups in the first embodiment. Map M in Figure 6 represents the three-dimensional space created when a predetermined space is measured in three dimensions, using the x and y axes of a two-dimensional coordinate system that indicates the horizontal direction. Map M includes multiple shooting locations. For the sake of explanation, only four shooting locations are shown in Map M in Figure 6: shooting location SP1 with the minimum value mx on the x-axis, shooting location SP2 with the maximum value Mx on the x-axis, shooting location SP3 with the minimum value my on the y-axis, and shooting location SP4 with the maximum value My on the y-axis.

[0050] The classification unit 120 calculates the length of the map M in the x-axis direction (Mx - mx) and calculates the number of sections in the x-axis direction Cx (= (Mx - mx) / Lx) based on the predetermined length Lx per section. Similarly, the classification unit 120 calculates the length of the map M in the y-axis direction (My - my) and calculates the number of sections in the y-axis direction Cy (= (My - my) / Ly) based on the predetermined length Ly per section. At this time, the total number of sections in section group SG is Cx × Cy. Therefore, the classification unit 120 classifies all 2D images into one of the Cx × Cy groups based on the coordinates (x, y) of the shooting position corresponding to the 2D image.

[0051] The classification unit 120 may also have a predetermined number of compartments in the x-axis direction and the number of compartments in the y-axis direction.

[0052] (Step ST202) The feature point extraction unit 130 extracts feature points from each of the 2D images.

[0053] (Step ST203) The unclassified list generation unit 140 generates an unclassified list of 2D images for each of the multiple groups. The unclassified list may be arranged in descending order of the number of feature points.

[0054] (Step ST204) The first processing unit 150 sorts the 2D images included in the unclassified list into an acceptance list and an exclusion list for each of the multiple groups. Hereafter, the process in step ST204 will be referred to as the "sorting process". A specific example of the sorting process will be explained below using the flowchart in Figure 7.

[0055] Figure 7 is a flowchart showing a specific example of the sorting process in the flowchart of Figure 6. The flowchart in Figure 7 transitions from step ST203 in Figure 5 and starts from step ST301.

[0056] (Step ST301) The first processing unit 150 selects group Gn from a plurality of groups. n represents, for example, the group number. For example, the first processing unit 150 selects the groups in ascending order of their group numbers.

[0057] (Step ST302) The first processing unit 150 selects an arbitrary image IM from the unclassified list U_Gn of group Gn. The arbitrary image IM is, for example, the image with the most feature points.

[0058] (Step ST303) The first processing unit 150 compares the image IM with other image IOs included in the undiscriminated list U_Gn.

[0059] (Step ST304) The first processing unit 150 determines whether the image I / O satisfies the comparison conditions. If it is determined that the image I / O satisfies the comparison conditions, the process proceeds to step ST305. If it is determined that the image I / O does not satisfy the comparison conditions, the process proceeds to step ST306.

[0060] (Step ST305) After it is determined that the image I / O satisfies the comparison conditions, the first processing unit 150 removes the image I / O from the undetermined list U_Gn and adds it to the exclusion list R_Gn.

[0061] (Step ST306) After it is determined in step ST304 that image IO does not satisfy the comparison conditions, or after transitioning from step ST305, the first processing unit 150 determines whether or not image IM has been compared with all other images. If it is determined that image IM has been compared with all other images, the process proceeds to step ST307. If it is determined that image IM has not been compared with all other images, the process returns to step ST303.

[0062] (Step ST307) After determining that image IM has been compared with all other images, the first processing unit 150 removes image IM from the undetermined list U_Gn and adds it to the accepted list A_Gn. Specific examples of the series of processes from step ST303 to step ST307 will be explained below using Figures 8 and 9.

[0063] Figure 8 illustrates multiple shooting positions included in a specific section of the section group in Figure 6. Figure 8 shows section S in Figure 6. For ease of explanation, only four shooting positions are shown in section S in Figure 8: shooting positions SP11, SP12, SP13, and SP14. At shooting position SP11, the -y direction is captured. Shooting position SP12 is located in the -x and -y directions relative to shooting position SP11. At shooting position SP12, the x direction is captured. Shooting position SP13 is located in the -y direction relative to shooting position SP12 and is located between shooting positions SP11 and SP12 in the x direction. At shooting position SP13, the -y direction is captured. Shooting position SP14 is located in the x and -y directions relative to shooting position SP11. At shooting position SP14, the shooting direction is the -x direction.

[0064] Figure 9 illustrates multiple camera images corresponding to multiple shooting positions in Figure 8. Figure 9 shows four camera images: camera image IM11, camera image IM12, camera image IM13, and camera image IM14. Camera image IM11 corresponds to shooting position SP11, camera image IM12 corresponds to shooting position SP12, camera image IM13 corresponds to shooting position SP13, and camera image IM14 corresponds to shooting position SP14. Of the four camera images, camera image IM11 has the largest number of feature points. In addition, a portion of the target being photographed is included in camera image IM11 in each of camera images IM12, IM13, and IM14.

[0065] The first processing unit 150 compares the camera image IM11 with another image (for example, camera image IM12). In this example, the first processing unit 150 determines that the comparison condition is met if the number of matching local features between the images being compared is equal to or greater than a threshold. The first processing unit 150 determines that camera image IM12 meets the comparison condition and adds camera image IM12 to the exclusion list. Similarly, the first processing unit 150 adds camera image IM13 and camera image IM14 to the exclusion list, respectively. After comparing camera image IM11 with all the other images, the first processing unit 150 adds camera image IM11 to the acceptance list.

[0066] (Step ST308) The first processing unit 150 determines whether or not an image is included in the unclassified list U_Gn. If it is determined that an image is included in the unclassified list U_Gn, the process returns to step ST302. If it is determined that an image is not included in the unclassified list U_Gn, the process proceeds to step ST309.

[0067] (Step ST309) After it is determined that no image is included in the unclassified list U_Gn, the first processing unit 150 outputs the adoption list A_Gn for group Gn.

[0068] (Step ST310) The first processing unit 150 determines whether or not there is a group to be processed. For example, the first processing unit 150 determines that a group that has not undergone the series of processes from step ST301 to step ST309 is a group to be processed. If it is determined that there is a group to be processed, the process returns to step ST301. If it is determined that there is no group to be processed, the flowchart in Figure 7 ends, and the process proceeds to step ST130 in Figure 4.

[0069] (Step ST130) The output unit 160 outputs an adoption list for all 2D images. After step ST130, the flowchart in Figure 4 ends.

[0070] In summary, the feature space generation device 100 according to the first embodiment can be understood as performing a process to exclude 2D images that depict the same location by using 2D images.

[0071] [Effect] According to the first embodiment, the feature space generation device acquires a plurality of 2D images taken of a predetermined space, 3D point cloud data obtained by three-dimensional measurement of the predetermined space, and parameter information including information on the shooting position and shooting orientation corresponding to each of the plurality of 2D images. The plurality of 2D images are classified into a plurality of groups relating to the shooting position or target in the predetermined space. Each of the 2D images classified into the plurality of groups is compared based on the similarity of global features and the number of matching local features, or based on the similarity of the camera lens direction among the shooting orientations and the number of matching local features, and is assigned to either a first list used for generating a feature space of the predetermined space or a second list not used for generating a feature space.

[0072] Therefore, the feature space generation device according to the first embodiment can automatically exclude unnecessary images from images that have been captured in excess of what is needed when generating a feature space. Furthermore, since the feature space generation device according to the first embodiment classifies the captured images into multiple groups and then excludes unnecessary images from each group, it can avoid mismatching of two images located far apart in a space where similar patterns are widespread, for example, and suppress the erroneous exclusion of images that are actually needed when generating a feature space.

[0073] <Second Embodiment> The feature space generation device according to the first embodiment reduces the number of 2D images used to generate the feature space by excluding 2D images other than those to be processed based on 2D image data. The feature space generation device according to the second embodiment adopts a portion of the excluded 2D images as the target for processing based on 3D point cloud data.

[0074] [Configuration] Figure 10 is a block diagram showing an example of the functional configuration of a feature space generation device according to the second embodiment. The feature space generation device 200 in Figure 10 includes an acquisition unit 210, a classification unit 220, a feature point extraction unit 230, an unclassified list generation unit 240, a first processing unit 250, a visible point cloud data generation unit 260, a missing voxel extraction unit 270, a second processing unit 280, and an output unit 290.

[0075] Note that the acquisition unit 210, classification unit 220, feature point extraction unit 230, unclassified list generation unit 240, and first processing unit 250 are substantially the same as the acquisition unit 110, classification unit 120, feature point extraction unit 130, unclassified list generation unit 140, and first processing unit 150 of the first embodiment, so their description is omitted.

[0076] The feature space generation device 200 has substantially the same configuration as the hardware configuration shown in Figure 2 described in the first embodiment. Therefore, the feature space generation device 200 according to the second embodiment can function as a computer comprising an acquisition unit 210, a classification unit 220, a feature point extraction unit 230, an unclassified list generation unit 240, a first processing unit 250, a visible point cloud data generation unit 260, a missing voxel extraction unit 270, a second processing unit 280, and an output unit 290, similar to the description in the first embodiment.

[0077] The visible point cloud data generation unit 260 generates a single visible point cloud data for each image included in the adoption list. Specifically, the visible point cloud data generation unit 260 identifies the viewpoint corresponding to the image to be processed by referring to the parameter information corresponding to the image to be processed. The visible point cloud data generation unit 260 performs visible point determination of the 3D point cloud data based on the identified viewpoint and generates a single visible point cloud data by extracting the point cloud visible from the identified viewpoint.

[0078] For determining the visible points described above, methods such as those found in Reference 1 (S. Katz, A. Tal, and R. Basri, “Direct visibility of point sets”, ACM Transactions on Graphics, Vol. 26, No. 3, Article 24, July 2007.) can be used.

[0079] Furthermore, the visible point cloud data generation unit 260 generates integrated visible point cloud data by integrating all the generated single visible point cloud data.

[0080] The missing voxel extraction unit 270 detects the difference between the 3D point cloud data and the integrated visible point cloud data and extracts spatially missing voxel data. Spatially missing voxel data is a set of element voxels that contain point clouds that exist in the 3D point cloud data but not in the integrated visible point cloud data.

[0081] For the above difference detection, methods such as those described in Reference 2 (D. Girardeau-Montaut, et.al, “Change detection on point cloud data acquired with a ground laser scanner”, ISPRS WG III / 3, III / 4, V / 3 Workshop Laser scanning 2005, ISPRS Archives - Volume XXXVI-3 / W19, 2005, pp.30-35, September 12-14, 2005) can be used.

[0082] The second processing unit 280 removes 2D images included in the exclusion list that meet predetermined conditions from the exclusion list and adds them to the acceptance list. Specifically, the second processing unit 280 performs ray tracing on each image included in the exclusion list based on the shooting position and shooting orientation. For images that intersect with spatially deficient voxel data through ray tracing, the second processing unit 280 removes the image that intersects with the most element voxels from the exclusion list and adds it to the acceptance list. The second processing unit 280 excludes the element voxels that intersect with the images added to the acceptance list from the spatially deficient voxel data. The second processing unit repeatedly performs this exclusion process until the spatially deficient voxel data no longer meets the spatially deficient condition or until there are no more images that intersect with the spatially deficient voxel data.

[0083] As described above, the second processing unit 280 may determine whether or not the spatial missing voxel data satisfies the spatial missing condition. The spatial missing condition is based, for example, on the number of points contained within the element voxel. Specifically, the second processing unit 280 determines that the spatial missing condition is satisfied if the number of points in the 3D point cloud data contained within the extracted element voxel is equal to or greater than a predetermined threshold (the fourth threshold). In other words, spatial missing voxel data that satisfies the spatial missing condition means that there is a missing value in the integrated visible point cloud data that needs to be filled in.

[0084] The output unit 290 outputs an acceptance list for all 2D images, including the 2D images that have been moved from the exclusion list to the acceptance list by the second processing unit 280.

[0085] The configuration of the feature space generation device according to the second embodiment has been described above. Next, the operation of the feature space generation device according to the second embodiment will be described.

[0086] [Operation] Figure 11 is a flowchart showing an example of list output processing in a feature space generation device according to the second embodiment. The flowchart in Figure 11 is started, for example, when a user executes the list output processing.

[0087] (Step ST410) The acquisition unit 210 acquires multiple 2D images, 3D point cloud data, and parameter information.

[0088] (Step ST420) The feature space generation device 200 excludes 2D images other than those to be processed based on a plurality of 2D images. Hereafter, the process in step ST420 will be referred to as the "exclusion process based on 2D images". Note that specific examples of the exclusion process based on 2D images are substantially the same as the flowcharts in Figures 5 and 7 described in the first embodiment, so their explanation will be omitted.

[0089] (Step ST430) The feature space generation device 200 selects a portion of the excluded 2D images as the 2D images to be processed based on the multiple 2D images and 3D point cloud data. Hereafter, the process in step ST430 will be referred to as the "exclusion removal process". A specific example of the exclusion removal process will be explained below using the flowchart in Figure 12.

[0090] Figure 12 is a flowchart showing a specific example of the exclusion removal process in the flowchart of Figure 11. The flowchart in Figure 12 transitions from step ST420 in Figure 11 and starts from step ST501.

[0091] (Step ST501) The visible point cloud data generation unit 260 generates a single visible point cloud data for each image included in the adoption list.

[0092] (Step ST502) The visible point cloud data generation unit 260 generates integrated visible point cloud data by integrating all the individual visible point cloud data.

[0093] (Step ST503) The missing voxel extraction unit 270 detects the difference between the 3D point cloud data and the integrated visible point cloud data and extracts spatially missing voxel data. The spatial missing data will be explained below using Figure 13.

[0094] Figure 13 illustrates an example of spatial loss in the second embodiment. Figure 13 shows point cloud data PC11 as an example of integrated visible point cloud data. Point cloud data PC11 includes a missing area DA11. The missing area DA11 corresponds to spatial loss voxel data.

[0095] (Step ST504) The second processing unit 280 determines whether the spatially missing voxel data satisfies the spatially missing condition. If it is determined that the spatially missing voxel data satisfies the spatially missing condition, the process proceeds to step ST505. If it is determined that the spatially missing voxel data does not satisfy the spatially missing condition, the flowchart in Figure 12 ends, and the process proceeds to step ST440 in Figure 11.

[0096] (Step ST505) After it is determined that the spatially missing voxel data satisfies the spatially missing condition, the second processing unit 280 performs ray tracing based on the shooting position and shooting orientation for each image included in the exclusion list.

[0097] (Step ST506) The second processing unit 280 determines whether or not there are images that intersect with spatially missing voxel data by ray tracing. If it is determined that there are images that intersect with spatially missing voxel data, the process proceeds to step ST507. If it is determined that there are no images that intersect with spatially missing voxel data, the flowchart in Figure 12 ends, and the process proceeds to step ST440 in Figure 11.

[0098] (Step ST507) After determining that there are images that intersect with spatially missing voxel data, the second processing unit 280 removes the image that intersects with the most element voxels from the exclusion list and adds it to the acceptance list. A specific example of the series of processes from step ST505 to ST507 will be explained below with reference to Figure 14.

[0099] Figure 14 illustrates an example of the camera image being revived and the reduction of spatial loss in the second embodiment. Figure 14 shows the camera image IM21 that was on the exclusion list and the point cloud data PC21 when the camera image IM21 is used. Note that the point cloud data PC21 shows the same field of view as the point cloud data PC11 in Figure 13.

[0100] The second processing unit 280 performs ray tracing on the camera image IM21 based on the shooting position and orientation. The second processing unit 280 determines that the camera image IM21 is an image that intersects with spatially missing voxel data (for example, corresponding to the missing area DA11 in Figure 13) by ray tracing. The second processing unit 280 removes the camera image IM21 from the exclusion list and adds it to the acceptance list. As a result of these processes, the point cloud data PC21 will include a cover area CA21 that covers the missing area DA11 in the point cloud data PC11.

[0101] (Step ST508) The second processing unit 280 removes intersecting element voxels with respect to the images added to the adoption list from the spatially missing voxel data.

[0102] (Step ST509) The second processing unit 280 determines whether the spatially missing voxel data satisfies the spatially missing condition. If it is determined that the spatially missing voxel data satisfies the spatially missing condition, the process returns to step ST506. If it is determined that the spatially missing voxel data does not satisfy the spatially missing condition, the flowchart in Figure 12 ends, and the process proceeds to step ST440 in Figure 11.

[0103] (Step ST440) The output unit 290 outputs an adoption list for all 2D images. After step ST440, the flowchart in Figure 11 ends.

[0104] In summary, the feature space generation device 200 according to the second embodiment can be understood as performing a process to confirm the coverage rate of point clouds in 3D space that cannot be determined by 2D images alone, using point cloud data, and to make the entire 3D space comprehensively covered.

[0105] [Effect] According to the second embodiment, the feature space generation device acquires a plurality of 2D images taken of a predetermined space, 3D point cloud data obtained by three-dimensional measurement of the predetermined space, and parameter information including information on the shooting position and shooting orientation corresponding to each of the plurality of 2D images. The plurality of 2D images are classified into a plurality of groups relating to the shooting position or target in the predetermined space. Each of the 2D images classified into the plurality of groups is compared based on the similarity of global features and the number of matching local features, or based on the similarity of the camera lens direction among the shooting orientations and the number of matching local features, and is assigned to either a first list used for generating a feature space of the predetermined space or a second list not used for generating a feature space.

[0106] Therefore, the feature space generation device according to the second embodiment can obtain the same effects as the first embodiment.

[0107] Furthermore, the feature space generation device according to the second embodiment generates single visible point cloud data for point clouds identified from viewpoints corresponding to each 2D image included in the first list for each of the multiple groups, based on the shooting position, shooting orientation, and 3D point cloud data. It generates integrated visible point cloud data by integrating all the single visible point cloud data. It performs difference detection between the 3D point cloud data and the integrated visible point cloud data. It extracts a set of voxels containing point clouds that exist in the 3D point cloud data but not in the integrated visible point cloud data as spatially missing voxel data. It performs ray tracing based on the shooting position and shooting orientation for each 2D image included in the second list. It removes 2D images that intersect with the spatially missing voxel data from the second list and adds them to the first list.

[0108] As a result, the feature space generation device according to the second embodiment can suppress missing data in the 3D space and generate a feature space that comprehensively includes image features in the space by performing exclusion processing based on 3D point cloud data after exclusion processing based on captured images. If there are missing data in the feature space, the estimation of the shooting position and orientation of the camera image captured in that missing space will fail. However, if there are few missing data in the feature space, the failure to estimate the shooting position and orientation of the camera image can be suppressed, enabling robust estimation.

[0109] <Third Embodiment> The feature space generation device according to the first embodiment reduces the number of 2D images used to generate the feature space by excluding 2D images other than those to be processed based on 2D image data. On the other hand, the feature space generation device according to the third embodiment reduces the number of 2D images used to generate the feature space by excluding 2D images other than those to be processed based on 3D point cloud data.

[0110] [Configuration] Figure 15 is a block diagram showing an example of the functional configuration of the feature space generation device according to the third embodiment. The feature space generation device 300 in Figure 15 includes an acquisition unit 310, a feature point cloud data generation unit 320, a feature missing voxel extraction unit 330, a third processing unit 340, and an output unit 350. Note that the acquisition unit 310 is substantially the same as the acquisition unit 110 of the first embodiment, so its description is omitted. In addition, the "unclassified list" in the third embodiment represents a list of 2D images that have not been assigned to the acceptance list or the exclusion list.

[0111] The feature space generation device 300 has substantially the same configuration as the hardware configuration shown in Figure 2 described in the first embodiment. Therefore, the feature space generation device 300 according to the third embodiment can function as a computer comprising an acquisition unit 310, a feature point cloud data generation unit 320, a feature missing voxel extraction unit 330, a third processing unit 340, and an output unit 350, similar to the description in the first embodiment.

[0112] The feature point cloud data generation unit 320 generates single local feature point cloud data for each image included in the unclassified list. Specifically, the feature point cloud data generation unit 320 identifies the viewpoint corresponding to the image to be processed by referring to the parameter information corresponding to the image to be processed. The feature point cloud data generation unit 320 performs a projection transformation of the 3D point cloud data based on the identified viewpoint and generates single local feature point cloud data relating to the local features of the point cloud identified from the viewpoint corresponding to each of the multiple images. In other words, the single local feature point cloud data corresponds to the single visible point cloud data in the second embodiment with multiple local features associated with it. Another way to express this is that the single local feature point cloud data corresponds to the data in the second embodiment where only the point cloud associated with local features remains from the single visible point cloud data (this is a part of the point cloud of the single visible point cloud data).

[0113] Furthermore, the feature point cloud data generation unit 320 selects one arbitrary image from the images included in the unclassified list and generates integrated local feature point cloud data by integrating all single local feature point cloud data (target point cloud data) excluding the single local feature point cloud data (target point cloud data) of the selected arbitrary image. The feature point cloud data generation unit 320 may also select the image with the fewest local features as the arbitrary image. In addition, the feature point cloud data generation unit 320 may sort the unclassified list according to predetermined conditions and select the sorted images in order, or select them randomly.

[0114] The feature-deficient voxel extraction unit 330 detects the difference between the integrated local feature point cloud data and the target point cloud data and extracts feature-deficient voxel data. Feature-deficient voxel data is a set of voxels containing point clouds with local features (local feature point clouds) that exist in the target point cloud data but not in the integrated feature point cloud data. The difference detection described above can be performed using the methods described in Reference 2, etc.

[0115] The third processing unit 340 sorts the 2D images included in the unclassified list into an accept list and an exclusion list. Specifically, if the feature-deficient voxel data satisfies the feature-deficient condition, the third processing unit 340 removes the image of the target point cloud data used in difference detection from the unclassified list and adds it to the accept list. If the feature-deficient voxel data does not satisfy the feature-deficient condition, the third processing unit 340 removes the image of the target point cloud data used in difference detection from the unclassified list and adds it to the exclusion list. The third processing unit 340 repeatedly performs this sorting process until all images included in the unclassified list are sorted into the accept list and the exclusion list.

[0116] The above feature loss condition is based, for example, on the number of local feature point clouds contained within an element voxel. Specifically, the third processing unit 340 determines that the feature loss condition is met if the number of local feature point clouds contained within the extracted element voxel is equal to or greater than a predetermined threshold (the fifth threshold). In other words, feature loss voxel data that satisfies the feature loss condition means that there are missing values ​​in the integrated feature point cloud data that need to be filled in.

[0117] The output unit 350 outputs an adoption list for all 2D images that have been sorted by the third processing unit 340.

[0118] The configuration of the feature space generation apparatus according to the third embodiment has been described above. Next, the operation of the feature space generation apparatus according to the third embodiment will be described.

[0119] [Operation] Figure 16 is a flowchart showing an example of list output processing in a feature space generation device according to the third embodiment. The flowchart in Figure 16 is started, for example, when the user executes the list output processing.

[0120] (Step ST610) The acquisition unit 310 acquires multiple 2D images, 3D point cloud data, and parameter information.

[0121] (Step ST620) The feature space generation device 300 excludes 2D images other than those to be processed based on the 3D point cloud data. Hereafter, the process in step ST620 will be referred to as the "exclusion process based on 3D point cloud data". A specific example of the exclusion process based on 3D point cloud data will be explained below using the flowchart in Figure 17.

[0122] Figure 17 is a flowchart showing a specific example of exclusion processing based on 3D point cloud data in the flowchart of Figure 16. The flowchart of Figure 17 transitions from step ST610 in Figure 16 and starts from step ST701.

[0123] (Step ST701) The feature point cloud data generation unit 320 generates single local feature point cloud data for each image included in the unclassified list.

[0124] (Step ST702) The feature point cloud data generation unit 320 selects an arbitrary image Im from the images included in the unclassified list. The arbitrary image is, for example, an image selected randomly.

[0125] (Step ST703) The feature point cloud data generation unit 320 generates integrated local feature point cloud data by integrating all single local feature point cloud data except for the target point cloud data of image Im.

[0126] (Step ST704) The feature-deficient voxel extraction unit 330 detects the difference between the integrated local feature point cloud data and the target point cloud data of image Im, and extracts feature-deficient voxel data.

[0127] (Step ST705) The third processing unit 340 determines whether the feature-missing voxel data satisfies the feature-missing condition. If it is determined that the feature-missing voxel data satisfies the feature-missing condition, the process proceeds to step ST706. If it is determined that the feature-missing voxel data does not satisfy the feature-missing condition, the process proceeds to step ST707.

[0128] (Step ST706) After it is determined that the feature-missing voxel data satisfies the feature-missing condition, the third processing unit 340 removes image Im from the unclassified list and adds it to the accepted list. After step ST706, the process proceeds to step ST708.

[0129] (Step ST707) After it is determined that the feature-missing voxel data does not satisfy the feature-missing condition, the third processing unit 340 removes image Im from the unclassified list and adds it to the exclusion list. After step ST707, the process proceeds to step ST708.

[0130] (Step ST708) The third processing unit 340 determines whether or not an image is included in the unclassified list. If it is determined that an image is included in the unclassified list, the process returns to step ST702. If it is determined that an image is not included in the unclassified list, the flowchart in Figure 17 ends, and the process proceeds to step ST630 in Figure 16.

[0131] (Step ST630) The output unit 350 outputs an adoption list for all 2D images. After step ST630, the flowchart in Figure 16 ends.

[0132] In summary, the feature space generation device 300 according to the third embodiment can be understood as performing a process to confirm the coverage rate of feature points in 3D space using point cloud data and to prevent the loss of feature points in 3D space.

[0133] [Effect] According to the third embodiment, the feature space generation device acquires a plurality of 2D images taken of a predetermined space, 3D point cloud data obtained by three-dimensional measurement of the predetermined space, and parameter information including information on the shooting position and shooting orientation corresponding to each of the plurality of 2D images. For each of the plurality of 2D images, it performs a projection transformation of the 3D point cloud data based on the shooting position and shooting orientation in the predetermined space. It generates single local feature point cloud data relating to the local features of the point cloud identified from the viewpoint corresponding to each of the plurality of 2D images. It generates integrated local feature point cloud data by integrating all single local feature point cloud data except for the target point cloud data of an arbitrary target 2D image. It performs difference detection between the integrated local feature point cloud data and the target point cloud data. It extracts a set of voxels containing point clouds with local features that exist in the target point cloud data but not in the integrated local feature point cloud data as feature-deficient voxel data. Based on the feature-deficient voxel data, the target 2D images are sorted into either a first list used for generating a feature space of a predetermined space or a second list not used for generating a feature space.

[0134] Therefore, the feature space generation device according to the third embodiment can automatically exclude unnecessary images from images that have been captured in excess of what is needed when generating the feature space.

[0135] Furthermore, in the third embodiment of the feature space generation apparatus, the processing unit adds the target 2D image to the first list if the feature-deficient voxel data contains local features above a threshold, and adds the target 2D image to the second list if the feature-deficient voxel data contains local features below a threshold.

[0136] As a result, the feature space generation device according to the third embodiment can suppress missing data in the 3D space and generate a feature space that comprehensively includes image features in the space. If there are missing data in the feature space, the estimation of the shooting position and orientation of the camera image taken in that missing space will fail. However, if there are few missing data in the feature space, the failure to estimate the shooting position and orientation of the camera image can be suppressed, enabling robust estimation. Furthermore, if voxels containing point clouds with local features overlap in different 2D images, the feature space generation device according to the third embodiment can exclude unnecessary point clouds by excluding one of the 2D images, thereby reducing the memory capacity required for the feature space.

[0137] <Fourth Embodiment> The feature space generation device according to the fourth embodiment performs two processes: an exclusion process based on 2D images as described in the first embodiment, and an exclusion process based on 3D point cloud data as described in the third embodiment.

[0138] [Configuration] Figure 18 is a block diagram showing an example of the functional configuration of a feature space generation device according to the fourth embodiment. The feature space generation device 400 in Figure 18 includes an acquisition unit 410, a classification unit 420, a feature point extraction unit 430, an unclassified list generation unit 440, a first processing unit 450, a feature point cloud data generation unit 460, a feature missing voxel extraction unit 470, a third processing unit 480, and an output unit 490.

[0139] Note that the acquisition unit 410, classification unit 420, feature point extraction unit 430, unclassified list generation unit 440, and first processing unit 450 are substantially the same as the acquisition unit 110, classification unit 120, feature point extraction unit 130, unclassified list generation unit 140, and first processing unit 150 of the first embodiment, so their description is omitted. Also, the feature-deficient voxel extraction unit 470 is substantially the same as the feature-deficient voxel extraction unit 330 of the third embodiment, so its description is omitted.

[0140] The feature space generation device 400 has substantially the same configuration as the hardware configuration shown in Figure 2 described in the first embodiment. Therefore, the feature space generation device 400 according to the fourth embodiment can function as a computer comprising an acquisition unit 410, a classification unit 420, a feature point extraction unit 430, an unclassified list generation unit 440, a first processing unit 450, a feature point cloud data generation unit 460, a feature missing voxel extraction unit 470, a third processing unit 480, and an output unit 490, similar to the description in the first embodiment.

[0141] The feature point cloud data generation unit 460 generates single local feature point cloud data for each image included in the adoption list. The feature point cloud data generation unit 460 differs from the feature point cloud data generation unit 320 of the third embodiment in that it generates single local feature point cloud data for images included in the adoption list, rather than the unclassified list. Note that the other processing of the feature point cloud data generation unit 460 is substantially the same as that of the feature point cloud data generation unit 320, so its explanation is omitted.

[0142] The third processing unit 480 either retains the 2D images included in the acceptance list in the acceptance list or adds them to the exclusion list. Specifically, if the feature-deficient voxel data satisfies the feature-deficient condition, the third processing unit 480 retains the image of the target point cloud data used in difference detection in the acceptance list. If the feature-deficient voxel data does not satisfy the feature-deficient condition, the third processing unit 480 removes the image of the target point cloud data used in difference detection from the acceptance list and adds it to the exclusion list. The third processing unit 480 repeatedly performs this exclusion process until there are no more unprocessed images included in the acceptance list.

[0143] The output unit 490 outputs an adoption list for all 2D images, including those moved from the adoption list to the exclusion list by the third processing unit 480.

[0144] The configuration of the feature space generation apparatus according to the fourth embodiment has been described above. Next, the operation of the feature space generation apparatus according to the fourth embodiment will be described.

[0145] [Operation] Figure 19 is a flowchart showing an example of list output processing in a feature space generation device according to the fourth embodiment. The flowchart in Figure 19 is started, for example, by the user executing the list output processing.

[0146] (Step ST810) The acquisition unit 410 acquires multiple 2D images, 3D point cloud data, and parameter information.

[0147] (Step ST820) The feature space generation device 400 excludes 2D images other than those to be processed based on a plurality of 2D images. Hereafter, the process in step ST820 will be referred to as the "exclusion process based on 2D images". Note that a specific example of the exclusion process based on 2D images is substantially the same as the flowcharts in Figures 5 and 7 described in the first embodiment, so the explanation will be omitted.

[0148] (Step ST830) The feature space generation device 400 excludes 2D images other than those to be processed based on the 3D point cloud data. Hereafter, the process in step ST830 will be referred to as the "exclusion process based on 3D point cloud data". A specific example of the exclusion process based on 3D point cloud data will be explained below using the flowchart in Figure 20.

[0149] Figure 20 is a flowchart showing a specific example of exclusion processing based on 3D point cloud data in the flowchart of Figure 19. The flowchart in Figure 20 transitions from step ST820 in Figure 19 and starts from step ST901.

[0150] (Step ST901) The feature point cloud data generation unit 460 generates single local feature point cloud data for each image included in the adoption list.

[0151] (Step ST902) The feature point cloud data generation unit 460 selects an arbitrary image Im from the images included in the adoption list. The arbitrary image is, for example, an image selected randomly.

[0152] (Step ST903) The feature point cloud data generation unit 460 generates integrated local feature point cloud data by integrating all single local feature point cloud data except for the target point cloud data of image Im.

[0153] (Step ST904) The feature-deficient voxel extraction unit 470 detects the difference between the integrated local feature point cloud data and the target point cloud data of image Im, and extracts feature-deficient voxel data.

[0154] (Step ST905) The third processing unit 480 determines whether the feature-deficient voxel data satisfies the feature-deficient condition. If it is determined that the feature-deficient voxel data satisfies the feature-deficient condition, the third processing unit 480 adds image Im to the adoption list and proceeds to step ST907. If it is determined that the feature-deficient voxel data does not satisfy the feature-deficient condition, the process proceeds to step ST906.

[0155] (Step ST906) After it is determined that the feature-missing voxel data does not satisfy the feature-missing condition, the third processing unit 480 removes the image Im from the adoption list and adds it to the exclusion list. After step ST906, the process proceeds to step ST907.

[0156] (Step ST907) The third processing unit 480 determines whether or not an unprocessed image is included in the acceptance list. If it is determined that an unprocessed image is included in the acceptance list, the process returns to step ST902. If it is determined that no unprocessed image is included in the acceptance list, the flowchart in Figure 20 ends, and the process proceeds to step ST840 in Figure 19.

[0157] (Step ST840) The output unit 490 outputs an adoption list for all 2D images. After step ST840, the flowchart in Figure 19 ends.

[0158] [Effect] According to the fourth embodiment, the feature space generation device can exclude 2D images other than the target of processing based on a plurality of 2D images, and can also exclude 2D images other than the target of processing based on 3D point cloud data.

[0159] Therefore, the feature space generation device according to the fourth embodiment can obtain the effects of both the first and third embodiments.

[0160] <Fifth Embodiment> The feature space generation device according to the fifth embodiment performs three processes: an exclusion process based on 2D images as described in the first embodiment, an exclusion release process as described in the second embodiment, and an exclusion process based on 3D point cloud data as described in the third embodiment.

[0161] [Configuration] Figure 21 is a block diagram showing an example of the functional configuration of a feature space generation device according to the fifth embodiment. The feature space generation device 500 in Figure 21 includes an acquisition unit 501, a classification unit 502, a feature point extraction unit 503, an unclassified list generation unit 504, a first processing unit 505, a visible point cloud data generation unit 506, a missing voxel extraction unit 507, a second processing unit 508, a feature point cloud data generation unit 509, a feature missing voxel extraction unit 510, a third processing unit 511, and an output unit 512.

[0162] Note that the acquisition unit 501, classification unit 502, feature point extraction unit 503, unclassified list generation unit 504, and first processing unit 505 are substantially the same as the acquisition unit 110, classification unit 120, feature point extraction unit 130, unclassified list generation unit 140, and first processing unit 150 of the first embodiment, so their description is omitted. Also, the visible point cloud data generation unit 506, missing voxel extraction unit 507, and second processing unit 508 are substantially the same as the visible point cloud data generation unit 260, missing voxel extraction unit 270, and second processing unit 280 of the second embodiment, so their description is omitted. Furthermore, the feature point cloud data generation unit 509, the feature missing voxel extraction unit 510, the third processing unit 511, and the output unit 512 are substantially the same as the feature point cloud data generation unit 460, the feature missing voxel extraction unit 470, the third processing unit 480, and the output unit 490 of the fourth embodiment, so their description is omitted.

[0163] The configuration of the feature space generation apparatus according to the fifth embodiment has been described above. Next, the operation of the feature space generation apparatus according to the fifth embodiment will be described.

[0164] [Operation] Figure 22 is a flowchart showing an example of list output processing in a feature space generation device according to the fifth embodiment. The flowchart in Figure 22 is started, for example, by the user executing the list output processing.

[0165] (Step ST1010) The acquisition unit 501 acquires multiple 2D images, 3D point cloud data, and parameter information.

[0166] (Step ST1020) The feature space generation device 500 excludes 2D images other than those to be processed based on a plurality of 2D images. Hereafter, the process in step ST1020 will be referred to as the "exclusion process based on 2D images". Note that a specific example of the exclusion process based on 2D images is substantially the same as the flowcharts in Figures 5 and 7 described in the first embodiment, so the explanation will be omitted.

[0167] (Step ST1030) The feature space generation device 500 selects a portion of the excluded 2D images as the 2D images to be processed based on the multiple 2D images and 3D point cloud data. Hereafter, the processing in step ST1030 will be referred to as the "exclusion removal process". A specific example of the exclusion removal process is substantially the same as the flowchart in Figure 12 described in the second embodiment, so the explanation will be omitted.

[0168] (Step ST1040) The feature space generation device 500 excludes 2D images other than those to be processed based on the 3D point cloud data. Hereafter, the process in step ST1040 will be referred to as the "exclusion process based on 3D point cloud data". A specific example of the exclusion process based on 3D point cloud data is substantially the same as the flowchart in Figure 20 described in the fourth embodiment, so the explanation will be omitted.

[0169] (Step ST1050) The output unit 512 outputs an adoption list for all 2D images. After step ST1050, the flowchart in Figure 22 ends.

[0170] [Effect] According to the fifth embodiment, the feature space generation device can exclude 2D images other than those to be processed based on a plurality of 2D images, adopt a portion of the excluded 2D images as the 2D images to be processed based on the plurality of 2D images and 3D point cloud data, and exclude 2D images other than those to be processed based on 3D point cloud data.

[0171] Therefore, the feature space generation device according to the fifth embodiment can obtain the effects of both the second and third embodiments.

[0172] The flowcharts described in each of the above embodiments are examples. The steps in the flowcharts described in each embodiment may be rearranged as much as possible, and other steps may be added.

[0173] It should be noted that the present invention is not limited to the embodiments described above, and can be modified in various ways during implementation without departing from its essence. Furthermore, each embodiment may be combined as appropriate, and in that case, the combined effects can be obtained. Moreover, the above embodiments include various inventions, and various inventions can be extracted by selecting combinations from the multiple components disclosed. For example, if the problem can be solved and effects can be obtained even if some components are deleted from all the components shown in the embodiment, then the configuration with these components deleted can be extracted as an invention.

[0174] 100, 200, 300, 400, 500... Feature space generation device 101... Control circuit 102... Storage 103... Communication module 104... Interface 105... Drive 106... Storage medium 110, 210, 310, 410, 501... Acquisition unit 120, 220, 420, 502... Classification unit 130, 230, 430, 503... Feature point extraction unit 140, 240, 440, 504... Unclassified list generation unit 150, 250, 450, 505... First processing unit 160, 290, 350, 490, 512... Output unit 260, 506... Visible point cloud data generation unit 270, 507... Missing voxel extraction unit 280, 508... Second processing unit 320, 460, 509... Feature point cloud data generation unit 330, 470, 510... Feature missing voxel extraction unit 340, 480, 511... Third processing unit CA21... Coverage area DA11... Missing area IM11, IM12, IM13, IM14, IM21... Camera image M... Map PC11, PC21... Point cloud data S... Section SG... Section group SP1, SP2, SP3, SP4, SP11, SP12, SP13, SP14... Shooting location

Claims

1. An acquisition unit that acquires a plurality of 2D images taken of a predetermined space, 3D point cloud data obtained by three-dimensional measurement of the predetermined space, and parameter information including information on the shooting position and shooting orientation corresponding to each of the plurality of 2D images; a generation unit that performs a projection transformation of the 3D point cloud data based on the shooting position and shooting orientation in the predetermined space for each of the plurality of 2D images, generates single local feature point cloud data relating to the local features of the point cloud identified from the viewpoint corresponding to each of the plurality of 2D images, and generates integrated local feature point cloud data by integrating all of the single local feature point cloud data excluding the target point cloud data of an arbitrary target 2D image; an extraction unit that detects the difference between the integrated local feature point cloud data and the target point cloud data, and extracts a set of voxels containing point clouds having the local features that exist in the target point cloud data but do not exist in the integrated local feature point cloud data as feature-deficient voxel data. A feature space generation device comprising a processing unit that sorts the target 2D image into either a first list used for generating a feature space in a predetermined space or a second list not used for generating a feature space, based on the feature-deficient voxel data.

2. The feature space generation apparatus according to claim 1, wherein the processing unit adds the target 2D image to the first list if the feature-deficient voxel data contains local features equal to or greater than a threshold, and adds the target 2D image to the second list if the feature-deficient voxel data contains local features equal to or less than a threshold.

3. Obtain a plurality of 2D images taken of a predetermined space, 3D point cloud data obtained by three-dimensional measurement of the predetermined space, and parameter information including information on the shooting position and shooting orientation corresponding to each of the plurality of 2D images; perform a projection transformation of the 3D point cloud data based on the shooting position and shooting orientation in the predetermined space for each of the plurality of 2D images, generate single local feature point cloud data relating to the local features of the point cloud identified from the viewpoint corresponding to each of the plurality of 2D images, and generate integrated local feature point cloud data by integrating all of the single local feature point cloud data excluding the target point cloud data of an arbitrary target 2D image; perform difference detection between the integrated local feature point cloud data and the target point cloud data, and extract a set of voxels containing point clouds having the local features that exist in the target point cloud data but do not exist in the integrated local feature point cloud data as feature-deficient voxel data. A feature space generation method comprising: sorting the target 2D image into either a first list used for generating a feature space for a predetermined space or a second list not used for generating a feature space, based on the feature-deficient voxel data.

4. A program for causing a computer to function as a component of the feature space generation apparatus described in claim 1 or claim 2.

Images