Image management device, image management method, and program

Abstract

To reduce the burden in image management by automating association of images and extraction of a specific image.SOLUTION: An image management apparatus 10 includes: a three-dimensional point specifying unit 11 which specifies three-dimensional points corresponding to feature points extracted from a two-dimensional image obtained by imaging an object, out of three-dimensional points constituting three-dimensional point cloud data of the object; an image management unit 12 which associates identification information of the two-dimensional image from which the feature points have been extracted, with the three-dimensional points corresponding to the extracted feature points; a two-dimensional image specifying unit 13 which specifies, when a specific section of the object is designated, three-dimensional points corresponding to the designated section, and specifies identification information of a two-dimensional image associated with the specified three-dimensional points; and a display unit 14 which displays the two-dimensional image whose identification information has been specified on a screen.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image management device and an image management method for linking and managing a plurality of images, and further to a program for implementing these. [Background technology]

[0002] In recent years, the increase in vacant houses has become a social problem (see, for example, Non-Patent Document 1). This is a social problem because vacant houses are often left abandoned for many years, and are at risk of collapsing due to aging, making them dangerous. In addition, there is the possibility of arson by suspicious individuals, or suspicious individuals taking up residence in vacant houses without permission, and an increase in vacant houses leads to a deterioration of public safety.

[0003] To solve this problem of vacant houses, local governments and other organizations are conducting physical inspections of vacant houses. In physical inspections, on-site investigators photograph the vacant houses to understand their condition. Photographs are taken not only of the house as a whole, but also in detail. The numerous images taken are then linked to each vacant house and managed in a database or similar. Personnel review the images to determine the possibility of collapse, arson, suspicious people residing in the house, etc. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] "Basic Guidelines for Comprehensive and Planned Implementation of Measures Concerning Vacant Houses, etc.", [online], Ministry of Internal Affairs and Communications and Ministry of Land, Infrastructure, Transport and Tourism Notification No. 1, dated February 26, 2015 (Last revised Ministry of Internal Affairs and Communications and Ministry of Land, Infrastructure, Transport and Tourism Notification No. 1, dated June 30, 2021), [Retrieved July 8, 2021], Internet<URL:https: / / www.mlit.go.jp / jutakukentiku / house / content / 001411707.pdf> Summary of the Invention [Problem to be solved by the invention]

[0005] As mentioned above, images of vacant houses are linked and managed on a house-by-house basis, but the linking of images is done manually. Specifically, the image manager links each detailed image to the overall image using arrows or other methods on the screen of an application software such as a spreadsheet or word processor. This makes the task of linking images a burden for the manager. Furthermore, when a vacant house inspector wants to check the details of a specific vacant house, they must check the linking on the screen and identify the desired image. Retrieving such detailed images also places a burden on the inspector.

[0006] An example of an object of the present invention is to provide an image management device, an image management method, and a program that can reduce the burden of image management by automating the linking of multiple images and the retrieval of specific images. [Means for solving the problem]

[0007] In order to achieve the above object, an image management device according to one aspect of the present invention comprises: a three-dimensional point identification unit that identifies three-dimensional points corresponding to feature points extracted from a two-dimensional image obtained by photographing the object, among three-dimensional points that constitute the three-dimensional point cloud data of the object; an image management unit that associates identification information of the two-dimensional image from which the extracted feature points are extracted with the three-dimensional points corresponding to the extracted feature points; a two-dimensional image specifying unit that, when a specific portion of an object is specified, specifies the three-dimensional point corresponding to the specified portion, and further specifies identification information of the two-dimensional image associated with the specified three-dimensional point; a display unit that displays the two-dimensional image with the specified identification information on a screen; The present invention is characterized in that it is provided with:

[0008] In order to achieve the above object, an image management method according to one aspect of the present invention comprises: a three-dimensional point identification step of identifying three-dimensional points corresponding to feature points extracted from two-dimensional images obtained by photographing the object, among three-dimensional points constituting the three-dimensional point cloud data of the object; an image management step of associating identification information of the two-dimensional image from which the extracted feature points were extracted with the three-dimensional points corresponding to the extracted feature points; a two-dimensional image specifying step of, when a specific portion of the object is specified, specifying the three-dimensional point corresponding to the specified portion, and further specifying identification information of the two-dimensional image associated with the specified three-dimensional point; a display step of displaying the two-dimensional image with the specified identification information on a screen; The present invention is characterized by having the following:

[0009] Furthermore, in order to achieve the above object, a program according to one aspect of the present invention comprises: On the computer, a three-dimensional point identification step of identifying three-dimensional points corresponding to feature points extracted from two-dimensional images obtained by photographing the object, among three-dimensional points constituting the three-dimensional point cloud data of the object; an image management step of associating identification information of the two-dimensional image from which the extracted feature points were extracted with the three-dimensional points corresponding to the extracted feature points; a two-dimensional image specifying step of, when a specific portion of the object is specified, specifying the three-dimensional point corresponding to the specified portion, and further specifying identification information of the two-dimensional image associated with the specified three-dimensional point; a display step of displaying the two-dimensional image with the specified identification information on a screen; The method is characterized in that: [Effects of the Invention]

[0010] As described above, according to the present invention, it is possible to automate the linking of multiple images and the retrieval of specific images, thereby reducing the burden of image management. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of an image management device according to an embodiment. [Figure 2] FIG. 2 is a diagram specifically showing the configuration of the image management device according to the embodiment. [Figure 3] FIG. 3 is a diagram showing an example of frames constituting video data for generating three-dimensional point cloud data. [Figure 4] FIG. 4 is a diagram showing an example of a pair of images used to generate three-dimensional point cloud data. [Figure 5] FIG. 5 is a diagram showing an example of the three-dimensional coordinates and rotation matrix of the camera of the initial pair of images, which are determined from the camera matrix. [Figure 6] FIG. 6 is a diagram showing an example of a combination of an image newly selected after the selection of an initial pair of images and feature points extracted from the image. [Figure 7] FIG. 7 is a diagram showing an example of the correspondence between the three-dimensional points of the three-dimensional point cloud data and the feature points of the two-dimensional image data. [Figure 8] FIG. 8 is a diagram showing an example of the association data created from the correspondence shown in FIG. [Figure 9] FIG. 9 is a diagram illustrating the processing performed by the two-dimensional image specifying unit. [Figure 10] FIG. 10 is a flow diagram showing the process of generating three-dimensional point cloud data and associated data by the image management device according to the embodiment. [Figure 11] FIG. 11 is a flow diagram showing a two-dimensional image display process performed by the image management device according to the embodiment. [Figure 12] FIG. 12 is a diagram showing an example of association data used in the modified example. [Figure 13] FIG. 13 is a diagram illustrating an example of processing by a two-dimensional image specifying unit in the modified example. [Figure 14] FIG. 14 is a diagram illustrating another example of processing by the two-dimensional image specifying unit in the modified example. [Figure 15]FIG. 15 is a block diagram illustrating an example of a computer that realizes an image management device according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] (Embodiment) An image management device, an image management method, and a program according to an embodiment will be described below with reference to FIGS.

[0013] [Device configuration] First, the schematic configuration of an image management device in an embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing the schematic configuration of an image management device in an embodiment.

[0014] The image management device 10 shown in Fig. 1 is a device for linking and managing multiple images. As shown in Fig. 1, the image management device 10 includes a three-dimensional point identification unit 11, an image management unit 12, a two-dimensional image identification unit 13, and a display unit 14.

[0015] The 3D point identification unit 11 identifies 3D points corresponding to feature points extracted from 2D images obtained by photographing the object, among the 3D points (hereinafter referred to as "3D points") that constitute the 3D point cloud data of the object. Here, 3D point cloud data is data composed of a collection of feature points whose 3D coordinates have been calculated. In this specification, a 3D point means a feature point whose 3D coordinates have been calculated. The image management unit 12 associates the identification information of the 2D image from which the extracted feature points were extracted with the 3D points corresponding to the extracted feature points.

[0016] When a specific part of the object is specified, the 2D image specifying unit 13 specifies a 3D point corresponding to the specified part and further specifies identification information of the 2D image associated with the specified 3D point. The display unit 14 displays the 2D image with the specified identification information on the screen.

[0017] In this way, image management device 10 automatically links the 3D points that make up the 3D point cloud data of an object with the 2D image. Furthermore, when a specific part of the object is specified, image management device 10 automatically identifies the 2D image that corresponds to the specified part and displays the identified 2D image on the screen. In other words, image management device 10 automates the linking of multiple images and the retrieval of specific images, reducing the burden of image management.

[0018] Next, the configuration and functions of image management device 10 in the embodiment will be specifically described with reference to Figures 2 to 9. Figure 2 is a configuration diagram specifically showing the configuration of the image management device in the embodiment.

[0019] As shown in FIG. 2, in the embodiment, the image management device 10 includes, in addition to the above-mentioned three-dimensional point identification unit 11, image management unit 12, two-dimensional image identification unit 13, and display unit 14, a data acquisition unit 15, a feature point extraction unit 16, an input reception unit 17, and a memory unit 20.

[0020] The data acquisition unit 15 acquires video data 21 and two-dimensional image data 22 of the object, and stores the acquired video data 21 and two-dimensional image data 22 in the storage unit 20. The storage unit 20 is constructed by a storage area of ​​a storage device.

[0021] In this embodiment, the three-dimensional point identification unit 11 generates three-dimensional point cloud data of the object using each frame of the video data 21 acquired by the data acquisition unit 15, and stores the generated three-dimensional point cloud data 23 in the storage unit 20. The three-dimensional point cloud data may be generated using the video data 21 and the two-dimensional image data 22, or may be generated using only the two-dimensional image data 22.

[0022] The process of generating 3D point cloud data by the 3D point identification unit 11 will be specifically described with reference to Figs. 3 to 6. Fig. 3 is a diagram showing an example of frames constituting video data for generating 3D point cloud data. Fig. 4 is a diagram showing an example of paired images used for generating 3D point cloud data. Fig. 5 is a diagram showing an example of the 3D coordinates and rotation matrix of the camera of the initial paired images calculated from the camera matrix. Fig. 6 is a diagram showing an example of a combination of an image newly selected after selecting the initial paired images and feature points extracted from it.

[0023] As shown in Fig. 3, the three-dimensional point specifying unit 11 first selects an image 31 and an image 32 as a pair of images (initial pair of images) from among frames constituting the video data. The images 31 and 32 are two-dimensional images. In the example of Fig. 3, two images are illustrated, but the three-dimensional point specifying unit 11 can also select three or more images.

[0024] Next, the 3D point identification unit 11 calculates, for example, SIFT features or SURF features for each image to identify feature points, and further extracts corresponding feature points between the images as a combination of corresponding feature points. In Figure 3, the circled part is one of the feature points.

[0025] As shown in FIG. 4, feature points (m1 to m5) extracted from image 31 correspond to feature points (m'1 to m'5) extracted from image 32. m1 and m'1, m2 and m'2, m3 and m'3, m4 and m'4, and m5 and m'5 are combinations of feature points (hereinafter also referred to as "feature point pairs"). In the example of FIG. 4, image 31 is captured by camera 41, and image 32 is captured by camera 42. In FIG. 4, M (M1 to M5) are three-dimensional coordinates on the object corresponding to each feature point.

[0026] Next, the three-dimensional point identification unit 11 calculates the camera matrix P of the camera 41 that captured the image 31 and the camera matrix P' of the camera 42 that captured the image 32, using the feature point pairs (m1 to m5, m'1 to m'5) extracted from each of the initial pair images. Furthermore, when the position of the camera 41 is taken as the origin, the camera matrix P and the camera matrix P' can be expressed by the following equations 1 and 2, respectively.

[0027]

number

[0028]

number

[0029] In the above equation 1, I is the rotation matrix of the camera 41. As shown in FIG. 5, the position of the camera 41 is the origin, so I=(1,1,1). Also, in the above equation 2, R is the rotation matrix of the camera 42 (R=(R x ,R y ,R z )). As described above, t is a translation matrix, which corresponds to the three-dimensional coordinates of the position of the camera 42 (t=(t x ,t y ,t z )).

[0030] Therefore, in this case, R and t can be calculated by performing inverse calculations from the camera matrix P and the camera matrix P'. The three-dimensional point identification unit 11 calculates R and t by solving the equations shown in the following formulas 3 to 5 using the coordinates of each feature point. In formulas 3 to 5, m is the coordinate on image 31 obtained by normalizing m (m1 to m5). Similarly, m' is the coordinate on image 32 obtained by normalizing m' (m'1 to m'5). E is the essential matrix, and K is the camera calibration matrix.

[0031]

number

[0032]

number

[0033]

number

[0034] The calibration matrix K can be calculated from the following equations 6 and 7. x , c y is the center coordinate of the camera.

[0035]

number

[0036]

number

[0037] Next, the three-dimensional point specifying unit 11 calculates the three-dimensional coordinates M (M1 to M5) of the feature points by triangulation using the three-dimensional coordinates of the positions of the cameras and the rotation matrix.

[0038] Next, as shown in Fig. 6, the three-dimensional point identifying unit 11 newly selects one frame (image 33) from among the frames other than the initial pair of images that constitute the video data 21 and that are images from which feature points have been extracted, and sets the newly selected image 33 and one of the initial pair of images as a new pair of images. Image 33 is captured by camera 43. Note that in this example, cameras 41 to 43 are different from each other in terms of their relative positions with respect to the object, but are actually the same camera that captured the video.

[0039] Then, the three-dimensional point identification unit 11 identifies feature points (m''1 to m''3) of the image 33 that correspond to the feature points of the image 32, and sets the feature points of the image 32 and the feature points of the image 33 as feature point pairs. Then, the three-dimensional point identification unit 11 calculates a camera matrix Pn of the camera 43 that captured the image 33. The camera matrix Pn can be expressed by the following equation 8.

[0040]

number

[0041] Specifically, the three-dimensional point identification unit 11 calculates Rn and tn of the camera matrix Pn of the camera 43 by the PnP method using three-dimensional coordinates M1 to M3 corresponding to the two-dimensional coordinates in the image 33 of the identified feature points in the image 33.

[0042] Then, the 3D point identification unit 11 repeatedly executes the above-described process to calculate the 3D coordinates of each feature point of the object, and constructs 3D point cloud data 23 of the object from a set of feature points whose 3D coordinates have been calculated. The 3D point identification unit 11 also stores the constructed 3D point cloud data 23 in the storage unit 20.

[0043] The feature point extraction unit 16 reads out the two-dimensional image data 22 from the storage unit 20, and extracts feature points from each two-dimensional image based on the read two-dimensional image data. Specifically, like the above-mentioned three-dimensional point identification unit 11, the feature point extraction unit 16 calculates, for example, SIFT features or SURF features for each two-dimensional image to identify feature points, and extracts the identified feature points.

[0044] Furthermore, when feature point extraction unit 16 extracts feature points from the two-dimensional image, three-dimensional point identification unit 11 reads out three-dimensional point cloud data 23 stored in storage unit 20. Then, three-dimensional point identification unit 11 compares the set of three-dimensional points constituting the read out three-dimensional point cloud data 23 with the feature points extracted from the two-dimensional image by feature point extraction unit 16, with respect to the respective feature amounts. Then, for each extracted feature point, three-dimensional point identification unit 11 identifies a three-dimensional point whose degree of match of the feature amount is higher than a threshold value.

[0045] As described above, when the 3D point cloud data 23 is generated using the video data 21 and the 2D image data 22, or when the 3D point cloud data 23 is generated using only the 2D image data 22, the feature quantities of the 2D image are extracted by the 3D point identification unit 11 when the 3D point cloud data is generated. In these cases, the 3D point identification unit 11 identifies 3D points corresponding to the feature points of the 2D image when the 3D point cloud data is generated. Furthermore, in these cases, the image management device 10 does not include the feature point extraction unit 16.

[0046] In this embodiment, the image management unit 12 associates the identification information of the 2D image from which the extracted feature points were extracted with the 3D points identified as corresponding to the extracted feature points, and creates association data 24 indicating the association between them. The image management unit 12 also stores the created association data 24 in the storage unit 20.

[0047] Furthermore, the image management unit 12 can also construct a kd tree structure using three-dimensional points when storing the associated data 24 in the storage unit 20. In this manner, the speed of the matching process (search process) of the associated data 24 by the two-dimensional image specification unit 13, which will be described later, can be increased.

[0048] 7 and 8, the processing by the feature point extraction unit 16, the 3D point identification unit 11, and the image management unit 12 will be specifically described. Fig. 7 is a diagram showing an example of the correspondence between the 3D points of the 3D point cloud data and the feature points of the 2D image data. Fig. 8 is a diagram showing an example of associated data created from the correspondence shown in Fig. 7.

[0049] 7, the feature point extraction unit 16 reads out two-dimensional image data ID1 to ID3 from the storage unit 20 as two-dimensional image data 22. Then, the feature point extraction unit 16 extracts feature points a1, a2, and a3 from the two-dimensional image data ID1, extracts feature points a'1, a'2, and a'3 from the two-dimensional image data ID2, and further extracts feature points a''1, a''2, and a''3 from the two-dimensional image data ID3.

[0050] In the example of FIG. 7, the three-dimensional point identification unit 11 compares a set of three-dimensional points constituting the three-dimensional point cloud data 23 read from the storage unit 20 with the feature points (a1, a2, a3, a'1, a'2, a'3, a''1, a''2, a''3) extracted by the feature point extraction unit 16 with respect to each feature amount. Then, the three-dimensional point identification unit 11 identifies a three-dimensional point A1 corresponding to the feature points (a1, a'1, a''1) of the two-dimensional image data ID1. Furthermore, the three-dimensional point identification unit 11 identifies a three-dimensional point A2 corresponding to the feature points (a2, a'2, a''2) of the two-dimensional image data ID2, and identifies a three-dimensional point A3 corresponding to the feature points (a3, a'3, a''3) of the two-dimensional image data ID3.

[0051] When the three-dimensional point specifying unit 11 specifies three-dimensional points corresponding to the feature points of the two-dimensional image data, the image management unit 12 associates the three-dimensional points with the identification information (ID) of the two-dimensional image from which the feature points were extracted, as shown in Fig. 8, to create association data 24. In the example of Fig. 8, the three-dimensional points are associated with the identification information (ID) of the two-dimensional image data and the feature points of the two-dimensional image data.

[0052] 2, the display unit 14 is connected to the display device 50 and displays information on the screen of the display device 50. In the embodiment, the information may be displayed on the screen of a terminal device connected to the image management device 10. The display unit 14 displays not only the above-mentioned two-dimensional image but also the three-dimensional point cloud data 23 on the screen.

[0053] When the 3D point cloud data 23 is displayed on the screen by the display unit 14, in the embodiment, the user uses an input device to specify a specific part of the object displayed by the 3D point cloud data 23 on the screen. The input receiving unit 17 receives input of information about the specified part via the input device and sends the received information about the part to the 2D image specifying unit 13. Examples of the input device include a touch panel, a mouse, a keyboard, and a terminal device.

[0054] In this embodiment, the 2D image specification unit 13 first reads the 3D point cloud data 23 and the association data 24 from the storage unit 20. Then, when a specific portion of the object is specified, the 2D image specification unit 13 specifies a 3D point that exists in the specified portion. Specifically, when a point on the object is specified by clicking, for example, the 2D image specification unit 13 searches for 3D points within a set range centered on the point, and specifies the 3D point that is closest to the specified point among the searched 3D points. Next, the 2D image specification unit 13 compares the specified 3D point with the association data 24 to specify the ID of the 2D image data associated with the 3D point.

[0055] Thereafter, the 2D image specification unit 13 notifies the specified ID to the display unit 14. In response to this, the display unit 14 reads out the 2D image data corresponding to the notified ID from the storage unit 20, and displays the read out 2D image data on the screen of the display device 50.

[0056] The processing by the two-dimensional image specifying unit 13 will be specifically described with reference to Fig. 9. Fig. 9 is a diagram for explaining the processing by the two-dimensional image specifying unit.

[0057] Assume that the user designates a specific region on the three-dimensional point cloud data as shown in Fig. 9. In this case, the two-dimensional image specification unit 13 specifies a three-dimensional point A1 as a three-dimensional point existing in the designated region, and further collates the specified three-dimensional point A1 with the association data 24 (see Fig. 8).

[0058] 8, since the three-dimensional point A1 is associated with the two-dimensional image data ID1 to ID3, the two-dimensional image specification unit 13 specifies ID1 to ID3. As a result, the display unit 14 displays the two-dimensional images of the two-dimensional image data ID1 to ID3 on the screen.

[0059] [Device operation] Next, the operation of the image management device in the embodiment will be explained using Figures 10 and 11. In the following explanation, Figures 1 to 9 will be referenced as appropriate. Also, in the embodiment, the image management method is implemented by operating the image management device. Therefore, the explanation of the image management method in the embodiment will be replaced by the explanation of the operation of the image management device below.

[0060] First, the process of generating the three-dimensional point cloud data 23 and the associated data 24 will be described with reference to Fig. 10. Fig. 10 is a flow diagram showing the process of generating the three-dimensional point cloud data and the associated data by the image management device in the embodiment.

[0061] 10, first, the data acquisition unit 15 acquires video data 21 and two-dimensional image data 22 of an object (step A1). In step A1, the data acquisition unit 15 stores the acquired video data 21 and two-dimensional image data 22 in the storage unit 20.

[0062] Next, the three-dimensional point identification unit 11 generates three-dimensional point cloud data 23 of the object using each frame of the video data 21 acquired in step A1, as shown in Figures 3 to 6 (step A2). In step A2, the three-dimensional point identification unit 11 stores the generated three-dimensional point cloud data 23 in the storage unit 20.

[0063] Next, as shown in FIG. 7, the feature point extraction unit 16 extracts feature points from each of the two-dimensional images based on the two-dimensional image data 22 acquired in step A1 (step A3).

[0064] Next, the 3D point identification unit 11 compares the set of 3D points constituting the 3D point cloud data 23 generated in step A2 with the feature points extracted in step A3 with respect to their respective feature amounts. Then, as shown in Fig. 7, the 3D point identification unit 11 identifies 3D points corresponding to the feature points extracted in step A3 based on the comparison result (step A4).

[0065] As described above, it is assumed that the 3D point cloud data 23 is generated using the video data 21 and the 2D image data 22, or the 3D point cloud data 23 is generated using only the 2D image data 22. In this case, the process of extracting feature amounts of the 2D image (step A3) and the process of identifying 3D points corresponding to the extracted feature points (step A4) are executed in the process of generating the 3D point cloud data in step A2. In these cases, steps A3 and A4 are integrated into step A2. However, even in these cases, when new 2D image data is added, steps A3 and A4 are executed for the newly added 2D image data.

[0066] Next, the image management unit 12 associates the identification information of the 2D image from which the feature points extracted in step A3 were extracted with the 3D points identified in step A4, and creates association data 24 indicating the association between them (step A5). In step A5, the image management unit 12 stores the created association data 24 in the storage unit 20.

[0067] By executing steps A1 to A5, multiple images are automatically linked, reducing the burden on the user in managing images.

[0068] The process up to the generation of the three-dimensional point group data 23 and the associated data 24 will be described with reference to Fig. 11. Fig. 11 is a flow diagram showing the display process of a two-dimensional image by the image management device in the embodiment.

[0069] 11, first, the display unit 14 displays an object on the screen of the display device 50 using the three-dimensional point cloud data 23 (step B1). An example of the object is a house, as shown in FIG.

[0070] Next, when the user specifies a specific part of the object on the screen using the input device, input receiving unit 17 receives input of information about the specified part via the input device (step B2). In step B2, input receiving unit 17 sends the received information about the part to two-dimensional image specifying unit 13.

[0071] Next, after executing step B2, the two-dimensional image specifying unit 13 reads the three-dimensional point cloud data 23 and the association data 24 from the storage unit 20, and specifies three-dimensional points present in the specified region (step B3).

[0072] Next, the two-dimensional image specifying unit 13 compares the specified three-dimensional point with the association data 24 to specify the ID of the two-dimensional image data associated with the three-dimensional point (step B4). In step B4, the two-dimensional image specifying unit 13 notifies the display unit 14 of the specified ID.

[0073] Thereafter, the display unit 14 reads out the two-dimensional image data corresponding to the ID identified in step B4 from the storage unit 20, and displays the two-dimensional image of the read two-dimensional image data on the screen of the display device 50 (step B5).

[0074] By executing steps B1 to B5, the user can simply specify a specific part on the 3D point cloud data, and a 2D image of the specific part will be automatically displayed on the screen. The user does not need to manually trace the associations and check the 2D image.

[0075] As described above, in the embodiment, image management device 10 generates association data 24 that associates 3D point cloud data 23 and 2D image data 22 of an object, thereby automatically linking them. Furthermore, by using association data 24, when a specific part of an object is specified, image management device 10 can automatically display on the screen a 2D image corresponding to the specified part. Image management device 10 can automate the linking of multiple images and the retrieval of detailed images of the object, thereby reducing the burden of managing images of objects.

[0076] [Variations] Next, a modified example of the embodiment will be described with reference to Figs. 12 to 14. Fig. 12 is a diagram showing an example of association data used in the modified example. Fig. 13 is a diagram explaining an example of processing by the two-dimensional image specifying unit in the modified example. Fig. 14 is a diagram explaining another example of processing by the two-dimensional image specifying unit in the modified example.

[0077] In this modification, as shown in Fig. 12, the image management unit 12 also creates associating data (hereinafter referred to as "first associating data") that associates identification information (ID) of a two-dimensional image with three-dimensional points, as shown in Fig. 8. However, in this modification, in addition to the first associating data, the image management unit 12 also creates associating data (hereinafter referred to as "second associating data") that associates feature points with corresponding three-dimensional points for each two-dimensional image data.

[0078] Therefore, as shown in FIG. 13, in this modified example, when a two-dimensional image of two-dimensional image data is displayed, the user can use an input device to specify a specific part of the object on the two-dimensional image.

[0079] The two-dimensional image specification unit 13 specifies feature points present in a specified location. Specifically, when a point on an object is specified by clicking, for example, the two-dimensional image specification unit 13 searches for feature points in the two-dimensional image within a set range centered on the specified point, and specifies the feature point closest to the specified point among the searched feature points. Next, the two-dimensional image specification unit 13 compares the specified feature point with second association data to specify a corresponding three-dimensional point, and further compares the specified three-dimensional point with first association data to specify the ID of the two-dimensional image data associated with the three-dimensional point.

[0080] 13, a two-dimensional image of the two-dimensional image data ID2 is displayed on the screen of the display device 50, and a specific part is specified in this two-dimensional image. As a result, the two-dimensional image specification unit 13 specifies the feature point a'1, and further specifies the corresponding three-dimensional point A1. Then, the two-dimensional image specification unit 13 specifies ID1 and ID3 other than ID2 that are associated with the three-dimensional point A1. Thereafter, the display unit displays the two-dimensional images of the two-dimensional image data ID1 and ID3 on the screen of the display device 50.

[0081] In this modification, two-dimensional image data of a two-dimensional image showing the entire object can be prepared, and the first association data and the second association data can be created including this two-dimensional image data. In this case, as shown in Fig. 14, a two-dimensional image showing the entire object can be displayed on the screen, and the user can specify a specific part in this two-dimensional image. In this case, the user can easily check the details while checking the whole object.

[0082] When 3D point cloud data is displayed on a screen, it is difficult to clearly display details compared to 2D images. In contrast, according to this modification, the user can specify a specific part on the 2D image instead of the 3D point cloud data, making it easier for the user to specify the part.

[0083] [program] The program in the embodiment may be any program that causes a computer to execute steps A1 to A5 shown in Fig. 10 and steps B1 to B5 shown in Fig. 11. By installing and executing this program on a computer, the image management device 10 and image management method in the embodiment can be realized. In this case, the processor of the computer functions as a feature point extraction unit 16, a three-dimensional point identification unit 11, an image management unit 12, a two-dimensional image identification unit 13, a display unit 14, a data acquisition unit 15, and an input acceptance unit 17, and performs processing. Examples of the computer include a general-purpose PC, a smartphone, and a tablet terminal device.

[0084] In addition, in the embodiment, the memory unit 20 may be realized by storing the data files that constitute these in a memory device such as a hard disk provided in the computer, or may be realized by a memory device of another computer.

[0085] The program in the embodiment may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as one of the feature point extraction unit 16, the 3D point identification unit 11, the image management unit 12, the 2D image identification unit 13, the display unit 14, the data acquisition unit 15, and the input reception unit 17.

[0086] [Physical configuration] Here, a computer that realizes image management device 10 by executing a program in the embodiment will be described with reference to Fig. 15. Fig. 15 is a block diagram showing an example of a computer that realizes the image management device in the embodiment.

[0087] 15, a computer 110 includes a CPU (Central Processing Unit) 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader / writer 116, and a communication interface 117. These components are connected to each other via a bus 121 so as to be able to communicate data with each other.

[0088] Furthermore, the computer 110 may include a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array) in addition to or instead of the CPU 111. In this aspect, the GPU or FPGA can execute the programs in the embodiments.

[0089] The CPU 111 loads a program in the embodiment, which is composed of a group of codes and stored in the storage device 113, into the main memory 112 and executes each code in a predetermined order to perform various calculations. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory).

[0090] The program in the embodiment is provided in a state stored in a computer-readable recording medium 120. The program in the embodiment may be distributed over the Internet connected via the communication interface 117.

[0091] Specific examples of the storage device 113 include a hard disk drive and a semiconductor storage device such as a flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and a mouse. The display controller 115 is connected to a display device 119 and controls the display on the display device 119.

[0092] The data reader / writer 116 mediates data transmission between the CPU 111 and the recording medium 120, reads programs from the recording medium 120, and writes processing results from the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and other computers.

[0093] Specific examples of the recording medium 120 include general-purpose semiconductor storage devices such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), magnetic recording media such as flexible disks, or optical recording media such as CD-ROMs (Compact Disk Read Only Memory).

[0094] Image management device 10 in this embodiment can be realized not by a computer with a program installed, but by using hardware (for example, electronic circuits) corresponding to each part. Furthermore, image management device 10 may be realized in part by a program and the remaining part by hardware.

[0095] Some or all of the above-described embodiments can be expressed by (Supplementary Note 1) to (Supplementary Note 12) described below, but are not limited to the following descriptions.

[0096] (Appendix 1) a three-dimensional point identification unit that identifies three-dimensional points corresponding to feature points extracted from a two-dimensional image obtained by photographing the object, among three-dimensional points that constitute the three-dimensional point cloud data of the object; an image management unit that associates identification information of the two-dimensional image from which the extracted feature points are extracted with the three-dimensional points corresponding to the extracted feature points; a two-dimensional image specifying unit that, when a specific portion of an object is specified, specifies the three-dimensional point corresponding to the specified portion, and further specifies identification information of the two-dimensional image associated with the specified three-dimensional point; a display unit that displays the two-dimensional image with the specified identification information on a screen; Equipped with An image management device characterized by:

[0097] (Appendix 2) 10. The image management device of claim 1, When the specific portion is designated in the 3D point cloud data displayed on a screen, the 2D image designation unit identifies the 3D point corresponding to the designated portion, and then identifies identification information of the 2D image. An image management device characterized by:

[0098] (Appendix 3) 10. The image management device of claim 1, the image manager further associates feature points extracted from a particular image of the object with the corresponding three-dimensional points; When the specific part is specified in the specific image displayed on the screen, the two-dimensional image specifying unit specifies a feature point corresponding to the specified part, and further specifies the three-dimensional point corresponding to the specified feature point, and specifies identification information of the two-dimensional image. An image management device characterized by:

[0099] (Appendix 4) An image management device according to any one of Supplementary Notes 1 to 3, the image management unit stores the identification information and the three-dimensional points in a storage area of ​​a storage device in a state in which the identification information and the three-dimensional points are associated with each other and in such a way that a kd tree structure is constructed by the three-dimensional points; An image management device characterized by:

[0100] (Appendix 5) a three-dimensional point identification step of identifying three-dimensional points corresponding to feature points extracted from two-dimensional images obtained by photographing the object, among three-dimensional points constituting the three-dimensional point cloud data of the object; an image management step of associating identification information of the two-dimensional image from which the extracted feature points were extracted with the three-dimensional points corresponding to the extracted feature points; a two-dimensional image specifying step of, when a specific portion of the object is specified, specifying the three-dimensional point corresponding to the specified portion, and further specifying identification information of the two-dimensional image associated with the specified three-dimensional point; a display step of displaying the two-dimensional image with the specified identification information on a screen; having An image management method comprising:

[0101] (Appendix 6) 6. The image management method according to claim 5, further comprising: In the two-dimensional image specifying step, when the specific part is specified in the three-dimensional point cloud data displayed on the screen, the three-dimensional point corresponding to the specified part is specified, and identification information of the two-dimensional image is specified. Further comprising: An image management method comprising:

[0102] (Appendix 7) 6. The image management method according to claim 5, further comprising: In the image management step, feature points extracted from a particular image of the object are further associated with the corresponding three-dimensional points; In the two-dimensional image specifying step, when the specific part is specified in the specific image displayed on the screen, a feature point corresponding to the specified part is specified, and further, the three-dimensional point corresponding to the specified feature point is specified, and identification information of the two-dimensional image is specified. An image management method comprising:

[0103] (Appendix 8) An image management method according to any one of Supplementary Notes 5 to 7, In the image management step, the identification information and the three-dimensional points are stored in a storage area of ​​a storage device in a state in which the identification information and the three-dimensional points are associated with each other and a kd tree structure is constructed by the three-dimensional points. An image management method comprising:

[0104] (Appendix 9) On the computer, a three-dimensional point identification step of identifying three-dimensional points corresponding to feature points extracted from two-dimensional images obtained by photographing the object, among three-dimensional points constituting the three-dimensional point cloud data of the object; an image management step of associating identification information of the two-dimensional image from which the extracted feature points were extracted with the three-dimensional points corresponding to the extracted feature points; a two-dimensional image specifying step of, when a specific portion of the object is specified, specifying the three-dimensional point corresponding to the specified portion, and further specifying identification information of the two-dimensional image associated with the specified three-dimensional point; a display step of displaying the two-dimensional image with the specified identification information on a screen; A program that executes.

[0105] (Appendix 10) 10. The program of claim 9, In the two-dimensional image specifying step, when the specific part is specified in the three-dimensional point cloud data displayed on the screen, the three-dimensional point corresponding to the specified part is specified, and identification information of the two-dimensional image is specified. A program characterized by:

[0106] (Appendix 11) 10. The program of claim 9, In the image management step, feature points extracted from a particular image of the object are further associated with the corresponding three-dimensional points; In the two-dimensional image specifying step, when the specific part is specified in the specific image displayed on the screen, a feature point corresponding to the specified part is specified, and further, the three-dimensional point corresponding to the specified feature point is specified, and identification information of the two-dimensional image is specified. A program characterized by:

[0107] (Appendix 12) The program according to any one of Supplementary Notes 9 to 11, In the image management step, the identification information and the three-dimensional points are stored in a storage area of ​​a storage device in a state in which the identification information and the three-dimensional points are associated with each other and a kd tree structure is constructed by the three-dimensional points. A program characterized by: [Industrial Applicability]

[0108] As described above, according to the present invention, it is possible to automate the linking of multiple images and the retrieval of specific images, thereby reducing the burden of image management. The present invention is useful for various systems that require the management of a large number of images in association with each other. [Explanation of symbols]

[0109] 10 Image management device 11 3D point identification part 12 Image Management Department 13 2D image identification unit 14 Display section 15 Data Acquisition Section 16 Feature point extraction unit 17 Input reception section 20 Memory section 21 Video data 22 2D Image Data 23 3D point cloud data 24 Associated Data 31, 32, 33 Images (2D images) 41, 42, 43 Camera 50 Display device 110 Computer 111 CPU 112 main memory 113 Storage device 114 Input Interface 115 Display Controller 116 Data Reader / Writer 117 Communication Interface 118 Input Devices 119 Display Device 120 Recording Media 121 Bus

Claims

1. a three-dimensional point identification unit that identifies three-dimensional points corresponding to feature points extracted from a two-dimensional image obtained by photographing the object, among three-dimensional points that constitute the three-dimensional point cloud data of the object; creating first association data that associates identification information of the two-dimensional image from which the extracted feature points are extracted with the three-dimensional points corresponding to the extracted feature points; and an image management unit that creates second association data for each of the plurality of two-dimensional images, associating the feature points extracted from the two-dimensional images with the corresponding three-dimensional points; When a point on the object is designated in the two-dimensional image of the object displayed on the screen, a feature point of the two-dimensional image of the object displayed on the screen is searched for within a set range centered on the designated point, and the feature point closest to the designated point is identified; matching the identified feature points with the second association data to identify the corresponding three-dimensional points; and a two-dimensional image specifying unit that checks the specified three-dimensional point against the first association data to specify identification information of all the two-dimensional images that are associated with the specified three-dimensional point; a display unit that displays all of the two-dimensional images for which identification information has been specified on a screen; Equipped with the two-dimensional image includes a two-dimensional image showing the entire object, the image management unit creates the first association data and the second association data, including a two-dimensional image showing the entire object; A point on the object is designated in a two-dimensional image showing the entire object displayed on a screen. An image management device characterized by:

2. 2. The image management device according to claim 1, the image management unit stores the identification information and the three-dimensional points in a storage area of ​​a storage device in a state in which the identification information and the three-dimensional points are associated with each other and in such a way that a kd tree structure is constructed by the three-dimensional points; An image management device characterized by:

3. a three-dimensional point specifying step of specifying three-dimensional points corresponding to feature points extracted from two-dimensional images obtained by photographing the object, among three-dimensional points constituting the three-dimensional point cloud data of the object; creating first association data that associates identification information of the two-dimensional image from which the extracted feature points are extracted with the three-dimensional points corresponding to the extracted feature points; and an image management step of creating second association data for each of the plurality of two-dimensional images, which associates the feature points extracted from the two-dimensional images with the corresponding three-dimensional points; When a point on the object is designated in the two-dimensional image of the object displayed on the screen, a feature point of the two-dimensional image of the object displayed on the screen is searched for within a set range centered on the designated point, and the feature point closest to the designated point is identified; matching the identified feature points with the second association data to identify the corresponding three-dimensional points; and a two-dimensional image specifying step of comparing the specified three-dimensional point with the first association data to specify identification information of all the two-dimensional images associated with the specified three-dimensional point; a display step of displaying on a screen all of the two-dimensional images for which identification information has been specified; and the two-dimensional image includes a two-dimensional image showing the entire object, In the image management step, the first association data and the second association data are created, including a two-dimensional image showing the entire object; A point on the object is designated in a two-dimensional image showing the entire object displayed on a screen. An image management method comprising:

4. On the computer, a three-dimensional point specifying step of specifying three-dimensional points corresponding to feature points extracted from two-dimensional images obtained by photographing the object, among three-dimensional points constituting the three-dimensional point cloud data of the object; creating first association data that associates identification information of the two-dimensional image from which the extracted feature points are extracted with the three-dimensional points corresponding to the extracted feature points; and an image management step of creating second association data for each of the plurality of two-dimensional images, which associates the feature points extracted from the two-dimensional images with the corresponding three-dimensional points; When a point on the object is designated in the two-dimensional image of the object displayed on the screen, a feature point of the two-dimensional image of the object displayed on the screen is searched for within a set range centered on the designated point, and the feature point closest to the designated point is identified; matching the identified feature points with the second association data to identify the corresponding three-dimensional points; and a two-dimensional image specifying step of comparing the specified three-dimensional point with the first association data to specify identification information of all the two-dimensional images associated with the specified three-dimensional point; a display step of displaying on a screen all of the two-dimensional images for which identification information has been specified; Execute the two-dimensional image includes a two-dimensional image showing the entire object, In the image management step, the first association data and the second association data are created, including a two-dimensional image showing the entire object; A program in which a point on the object is specified in a two-dimensional image showing the entire object displayed on a screen.

Images