Image display device, method, and recording medium

Abstract

The present invention provides an image display device, method, and program that, when an arbitrary photographic image is selected from a set of photographic images obtained by photographing a subject such as a structure, can easily grasp which part of the subject the selected photographic image was taken of. An image list representing a set of photographic images obtained by changing the photographic position and photographic direction with respect to the subject and photographing the subject is displayed on a display unit, and a selection instruction for an arbitrary photographic image is received from the image list through a user operation. A three-dimensional model of the subject is displayed on the display unit, and a mark indicating the position of the photographic image for which the selection instruction was given is superimposed and displayed on the displayed three-dimensional model. Thereby, the user can easily grasp which part of the subject (three-dimensional model) the selected photographic image was taken of.

Description

Technical Field

[0001] The present invention relates to an image display device, method, and program, and particularly to a technique for easily grasping the relationship between a subject and a photographed image obtained by photographing a part of the subject. Background Art

[0002] Conventionally, an image management device has been proposed that can centrally acquire images from a large number of images obtained by photographing a structure, for each structural set of the structure (Patent Document 1).

[0003] The image management device described in Patent Document 1 displays a three-dimensional point group related to a structure or a three-dimensional model composed of a mesh generated based on the three-dimensional point group on a display unit. When a user specifies three-dimensional coordinates of the structure using the displayed three-dimensional point group or three-dimensional model, three-dimensional points corresponding to a segment, which is a structural set in the structure, are extracted from the three-dimensional point group based on the specified three-dimensional coordinates. Here, the segment corresponds to an inspection target part of the structure. For example, in the inspection of a bridge, it is a main girder or a bottom plate, etc.

[0004] The image management device acquires an image associated with the three-dimensional points corresponding to the extracted segment from an image storage unit that stores images of the structure, and displays the acquired image on the display unit.

[0005] Prior Art Documents

[0006] Patent Documents

[0007] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2017-130146 Summary of the Invention

[0008] Technical Problem to be Solved by the Invention

[0009] According to the image management device described in Patent Document 1, it is possible to determine a segment, which is a structural set in the structure (in the case of a bridge, an inspection target part such as a main girder or a bottom plate), based on the three-dimensional coordinates of the structure specified by the user, acquire an image associated with the segment, and present it to the user.

[0010] However, in the image management device described in Patent Document 1, it is impossible to grasp which part of the structure the arbitrary photographed image selected by the user from a large number of photographed image groups obtained by photographing the structure is an image of. This is because, when the user wants to centrally view images obtained by photographing the inspection target part of the structure, the image management device described in Patent Document 1 acquires a target image (an image of the inspection target part) from a large number of images obtained by photographing the structure and presents it to the user.

[0011] On the other hand, when the photographing range of a photographed image is small relative to a structure, since there are multiple similar photographed images, it is impossible for a user to compare the photographed image with the structure to understand which part of the structure the photographed image was taken of.

[0012] The present invention has been completed in view of such circumstances, and an object thereof is to provide an image display device, method, and program that can easily understand which part of a subject the selected photographed image was taken of when a user selects an arbitrary photographed image from a group of photographed images obtained by photographing a subject such as a structure.

[0013] Means for Solving the Technical Problem

[0014] To achieve the above object, an image display device according to one aspect of the present invention includes: a storage unit that stores a three-dimensional model of a subject and a group of photographed images obtained by photographing the subject while changing the photographing position and photographing direction with respect to the subject; a first display control unit that causes an image list representing the group of photographed images to be displayed on a display unit; a first operation unit that receives a selection instruction of an arbitrary photographed image from the image list; a position information acquisition unit that acquires position information on the three-dimensional model corresponding to the photographed image selected by the selection instruction; and a second display control unit that reads out the three-dimensional model stored in the storage unit and causes it to be displayed on the display unit, and the second display control unit causes an identifier indicating the position of the photographed image selected by the selection instruction to be superimposed and displayed on the three-dimensional model displayed on the display unit based on the position information on the three-dimensional model acquired by the position information acquisition unit.

[0015] According to one aspect of the present invention, when an image list representing a group of photographed images is displayed on a display unit and a selection instruction of an arbitrary photographed image is received from the image list by a user operation, a three-dimensional model of the subject is displayed on the display unit, and an identifier indicating the position of the photographed image selected by the selection instruction is superimposed and displayed on the displayed three-dimensional model.

[0016] Thereby, the user can easily understand which part of the subject (three-dimensional model) the selected photographed image was taken of.

[0017] In the image display device according to another aspect of the present invention, preferably, it includes: a condition setting unit that sets conditions indicating screening of photographed images extracted from a group of photographed images and / or rearrangement of the group of photographed images; and an image list creation unit that creates an image list representing the screened photographed images from the group of photographed images and / or an image list after rearranging the photographed images based on the conditions set by the condition setting unit, and a first display control unit updates the image list displayed on the display unit using the image list created by the image list creation unit. By using the image list generated in this way, the user can easily select a desired photographed image from the group of photographed images.

[0018] In the image display device according to still another aspect of the present invention, preferably, the items displayed in the image list include a reduced image of each photographed image in the group of photographed images, identification information for determining each photographed image, the alignment degree of the photographed image with respect to the subject, the distance of the photographed image from the subject, the sharpness of the photographed image, and information indicating whether it is associated with an inspection record, damage detection result, or repair record of the subject, and one or more of the photographing date and time.

[0019] In the image display device according to still another aspect of the present invention, preferably, a second display control unit causes a three-dimensional model with an identifier overlapped to be displayed in a first display area of the display unit, reads out the selected and indicated photographed image from the storage unit and causes it to be displayed in a second display area of the display unit.

[0020] In the image display device according to still another aspect of the present invention, preferably, the second display control unit enlarges, translates, or rotates the three-dimensional model displayed on the display unit based on the position information on the obtained three-dimensional model, making the identifier overlapped on the three-dimensional model easier to view.

[0021] In the image display device according to still another aspect of the present invention, preferably, it includes a second operation unit that accepts a view operation of the three-dimensional model displayed on the display unit, and the second display control unit causes the three-dimensional model to be displayed on the display unit based on the view operation accepted by the second operation unit.

[0022] In the image display device according to still another aspect of the present invention, preferably, the view operation is an operation of enlarging, reducing, translating, or rotating the three-dimensional model displayed on the display unit, and the second display control unit enlarges, reduces, translates, or rotates the three-dimensional model displayed on the display unit based on the view operation accepted by the second operation unit.

[0023] In the image display device according to another aspect of the present invention, the three-dimensional model is a model represented by a three-dimensional point group composed of three-dimensional information of a plurality of points on the surface of the subject, a model representing the surface of the subject by an aggregate of polygonal faces based on the three-dimensional point group, or a model in which a photographic image obtained by photographing the subject is texture-mapped onto the polygonal faces.

[0024] The image display method according to another aspect of the present invention includes: a step of preparing a storage unit that stores a three-dimensional model of a subject and a set of photographic images obtained by photographing the subject while changing the photographic position and the photographic direction with respect to the subject; a first display step in which a first display control unit causes an image list representing the set of photographic images to be displayed on the display unit; a step in which a first operation unit receives a selection instruction of an arbitrary photographic image from the image list; a step in which a position information acquisition unit acquires position information on the three-dimensional model corresponding to the photographic image selected and indicated; and a second display step in which a second display control unit reads out the three-dimensional model stored in the storage unit and causes it to be displayed on the display unit. In the second display step, based on the acquired position information on the three-dimensional model, an identifier indicating the position of the photographic image selected and indicated is overlapped and displayed on the three-dimensional model displayed on the display unit.

[0025] In the image display method according to another aspect of the present invention, preferably, it includes: a step of setting, by a condition setting unit, conditions indicating screening of photographic images extracted from the set of photographic images and / or rearrangement of the photographic images; and a step in which an image list production unit produces an image list representing the photographic images screened from the set of photographic images and / or an image list after rearrangement of the photographic images based on the set conditions. In the first display step, the image list displayed on the display unit is updated using the produced image list.

[0026] In the image display method according to another aspect of the present invention, preferably, items displayed in the image list include one or more of a reduced image of each photographic image in the set of photographic images, identification information for identifying each photographic image, the alignment degree of the photographic image with respect to the subject, the distance of the photographic image from the subject, the sharpness of the photographic image, information indicating whether it is associated with an inspection record, a damage detection result, or a repair record of the subject, and the photographic date and time.

[0027] In the image display method according to another aspect of the present invention, preferably, in the second display step, the three-dimensional model with the identifier overlapped is displayed in a first display area of the display unit, and the photographic image selected and indicated is read out from the storage unit and displayed in a second display area of the display unit.

[0028] In the image display method according to another aspect of the present invention, preferably, in the second display step, based on the position information on the acquired three-dimensional model, the three-dimensional model displayed on the display unit is enlarged, translated, or rotated so that the identification overlappingly displayed on the three-dimensional model is easier to view.

[0029] In the image display method according to another aspect of the present invention, preferably, it includes a step of accepting a view operation of the three-dimensional model displayed on the display unit by a second operation unit, and in the second display step, the three-dimensional model is displayed based on the accepted view operation.

[0030] In the image display method according to another aspect of the present invention, the view operation is an operation of enlarging, reducing, translating, or rotating the three-dimensional model displayed on the display unit, and in the second display step, based on the view operation, the three-dimensional model displayed on the display unit is enlarged, reduced, translated, or rotated.

[0031] In the image display method according to another aspect of the present invention, the three-dimensional model is a model represented by a three-dimensional point group composed of three-dimensional information of a plurality of points on the surface of the subject, a model representing the surface of the subject by an aggregate of polygonal faces based on the three-dimensional point group, or a model obtained by texture-mapping a photographed image of the subject onto the polygonal faces.

[0032] The invention according to another aspect is an image display program installed in a computer accessible to a storage unit that stores a three-dimensional model of a subject and a set of photographed images obtained by changing the photographing position and photographing direction with respect to the subject to photograph the subject. The image display program causes the computer to implement: a function of displaying an image list representing the set of photographed images on the display unit; a function of accepting a selection instruction of an arbitrary photographed image from the image list; a function of acquiring position information on the three-dimensional model corresponding to the photographed image indicated by the selection; and a function of reading out the three-dimensional model stored in the storage unit and displaying it on the display unit. In this function, based on the acquired position information on the three-dimensional model, an identification indicating the position of the photographed image indicated by the selection is overlappingly displayed on the three-dimensional model displayed on the display unit.

[0033] Advantages of the Invention

[0034] According to the present invention, when a user selects and indicates an arbitrary photographed image from a set of photographed images obtained by photographing a subject such as a structure, the three-dimensional model of the subject is displayed on the display unit, and an identification indicating the position of the photographed image indicated by the selection is overlappingly displayed on the displayed three-dimensional model. Therefore, it is possible to easily grasp which part of the subject the photographed image selected from the set of photographed images was taken of. Brief Description of the Drawings

[0035] Figure 1 It is a block diagram showing an example of the hardware structure of the image display device related to the present invention.

[0036] Figure 2 It is a conceptual diagram showing a group of photographed images, a three-dimensional model, and an image list stored in the storage unit.

[0037] Figure 3 It is a conceptual diagram showing the generation of a three-dimensional model and an image list based on a group of photographed images.

[0038] Figure 4 It is a functional block diagram showing each function of the CPU.

[0039] Figure 5 It is a diagram showing an example of a screen of a display unit that displays a three-dimensional model showing a panorama of a bridge as a subject.

[0040] Figure 6 It is a diagram showing an example of a screen of a display unit that displays a three-dimensional model of a bridge that moves stereoscopically by view operations such as zooming.

[0041] Figure 7 It is a diagram showing an example of an orthoimage corresponding to a cofferdam of a bottom plate that is one of the management zones of a bridge.

[0042] Figure 8 It is a diagram showing an example of a damage map corresponding to the cofferdam.

[0043] Figure 9 It is a diagram showing an example of an orthoimage overlaid with a damage map corresponding to the cofferdam.

[0044] Figure 10 It is a chart showing an example of a damage quantity table corresponding to the cofferdam.

[0045] Figure 11 It is a chart showing an example of an image list.

[0046] Figure 12 It is a flowchart showing a first embodiment of the image display method related to the present invention.

[0047] Figure 13 It shows Figure 12 A diagram of a subroutine showing a first example of the process of step S100 shown.

[0048] Figure 14 It shows Figure 12 A diagram of a subroutine showing a second example of the process of step S100 shown.

[0049] Figure 15 It shows Figure 12Diagram of the subroutine of the third example of the process of step S100 shown

[0050] Figure 16 It is a diagram showing an example of a screen of a display unit that displays a photographed image indicated by a selection indication and a three-dimensional model of a photographing range including the photographed image

[0051] Figure 17 It is a flowchart showing a second embodiment of the image display method according to the present invention

[0052] Figure 18 It is a diagram showing an example of a screen of a display unit that displays the best photographed image determined from a plurality of photographed images

[0053] Figure 19 It is a diagram showing another example of a screen of a display unit that displays the best photographed image determined from a plurality of photographed images

[0054] Figure 20 It is a diagram showing still another example of a screen of a display unit that displays the best photographed image and the like determined from a plurality of photographed images Detailed Embodiment

[0055] Hereinafter, preferred embodiments of the image display device, method, and program according to the present invention will be described with reference to the drawings

[0056] [Hardware Structure of Image Display Device]

[0057] Figure 1 It is a block diagram showing an example of the hardware structure of the image display device according to the present invention

[0058] As Figure 1 For the image display device 10 shown, a personal computer or a workstation can be used. The image display device 10 in this example mainly includes an image acquisition unit 12, a storage unit 16, an operation unit 18, a CPU (Central Processing Unit) 20, a RAM (Random Access Memory) 22, a ROM (Read Only Memory) 24, and a display control unit 26

[0059] The image acquisition unit 12 is equivalent to an input / output interface and acquires photographed images obtained by photographing a subject. The subject in this example is a structure such as a bridge or a tunnel to be inspected

[0060] The images acquired by the image acquisition unit 12 are, for example, a plurality of images (a set of photographed images) obtained by photographing a subject (a structure) using a drone (unmanned aerial vehicle) equipped with a photographing device. Preferably, the set of photographed images covers the entire structure, and adjacent photographed images overlap in about 80% of the range. Therefore, although it also depends on the size of the structure, the set of photographed images is 1000 or more.

[0061] The set of photographed images acquired by the image acquisition unit 12 is stored in the storage unit 16.

[0062] The storage unit 16 is a storage unit composed of a hard disk device, a flash memory, etc. As Figure 2 shown, a set of photographed images 16A, a three-dimensional model 16B of the structure, and an image list 16C are stored in the storage unit 16.

[0063] As Figure 3 shown in the conceptual diagram, the three-dimensional model 16B and the image list 16C can be generated based on the set of photographed images 16A. In addition, the details of the three-dimensional model 16B and the image list 16C will be described later.

[0064] In addition, an operating system, an image display program related to the present invention, and various parameters are stored in the storage unit 16.

[0065] The operation unit 18 includes a keyboard, a mouse, etc. that are wired or wirelessly connected to a computer. In this example, in addition to functioning as an operation unit that accepts normal operation inputs of the computer, it also functions as a first operation unit that accepts a selection instruction of an arbitrary photographed image from the image list through a user operation. In addition, it functions as a second operation unit that accepts a view operation including magnification of the three-dimensional model displayed on the screen of the display unit 30. Furthermore, it functions as a third operation unit that accepts position information indicating a position on the three-dimensional model displayed on the display unit 30 according to a user operation.

[0066] In addition, the operation unit 18 functions as an operation unit of a condition setting unit that sets conditions (screening conditions) for screening photographed images extracted from the set of photographed images and / or conditions (sorting conditions) indicating rearrangement of the set of photographed images.

[0067] The CPU 20 reads out various programs stored in the storage unit 16 or the ROM 24, etc., and uniformly controls each unit based on the input from the operation unit 18. In addition, by executing the image display program related to the present invention, as Figure 4 shown, the CPU 20 has functions such as a photographed image search unit 20A, a photographed image determination unit 20B, a display switching unit 20C, an image list creation unit 20D, a position information acquisition unit 20E, and a condition setting unit 20F.

[0068] The RAM 22 serves as a working area for the CPU 20 and as a storage unit for temporarily storing the read program or various data.

[0069] The display control unit 26 functions as a first display control unit for displaying the image list created by the image list creation unit 20D (CPU 20) on the display unit 30, and also functions as a second display control unit for reading out the three-dimensional model stored in the storage unit 16 and displaying it on the display unit 30. Based on the position information on the three-dimensional model acquired by the position information acquisition unit 20E, the second display control unit overlays and displays an identifier indicating the position of the photographed image selected and indicated by the operation unit 18 (first operation unit) on the three-dimensional model displayed on the display unit 30.

[0070] In addition, the display control unit 26 functions as a third display control unit for reading out the photographed image determined by the photographed image determination unit 20B (CPU 20) from the storage unit 16 and displaying the read photographed image on the display unit 30. Moreover, it is a part that creates display data for display on the display unit 30 and outputs it to the display unit 30. In this example, it functions as a fourth display control unit that reads out the three-dimensional model stored in the storage unit 16 and displays the three-dimensional model on the display unit 30, and magnifies and displays the three-dimensional model based on a view operation including magnification of the three-dimensional model performed by the operation unit 18.

[0071] The display unit 30 uses various monitors such as a liquid crystal monitor that can be connected to a computer, and displays various information such as a three-dimensional model, a photographed image, and an image list according to the display data input from the display control unit 26. In addition, it is used as a part of the user interface together with the operation unit 18.

[0072] In the image display device 10 having the above structure, through an instruction input from the operation unit 18, the CPU 20 reads out the image display program according to the present invention stored in the storage unit 16 or the ROM 24, executes the image display program, and thereby displays various information on the display unit 30 according to the operation of the operation unit 18.

[0073] The three-dimensional model includes a three-dimensional point group that extracts feature points between mutually overlapping photographed images of a photographed image group obtained by photographing a structure, and estimates the position and posture of the imaging device mounted on the unmanned aerial vehicle based on the extracted feature points. In addition, the three-dimensional positions of the feature points are simultaneously estimated according to the estimation results of the position and posture of the imaging device.

[0074] There is a Structure from Motion (SfM) method that tracks the movement of multiple feature points from a set of photographic images in which the photographic position of a camera device is moved by a drone, while estimating the three-dimensional structure (Structure) of a structure and the pose of the camera device (Motion). In recent years, an optimization algorithm called bundle adjustment has been developed, which can obtain highly accurate outputs.

[0075] In addition, parameters of the camera device (focal length, image size of the image sensor, pixel pitch, etc.) required when applying the SfM method can be used with the parameters stored in the storage unit 16. Additionally, since an absolute scale cannot be obtained in the SfM method, for example, by indicating a known size of the structure (distance between two points, etc.), an absolute scale (three-dimensional position) can be obtained.

[0076] Here, the three-dimensional model can be considered a model represented by a three-dimensional point cloud of multiple points on the surface of the structure, a model representing the surface of the structure with an aggregate of polygonal faces (e.g., triangular patches) based on the three-dimensional point cloud, or a model in which a photographic image (texture) obtained by photographing the structure is texture-mapped onto the polygonal faces. The three-dimensional model of the structure in this example is a model in which the photographic image is texture-mapped onto the polygonal faces.

[0077] However, regular inspections of the structure are carried out for each inspection unit of the parts and components constituting the structure, that is, for each management area.

[0078] Hereinafter, a bridge will be used as an example of the structure for explanation.

[0079] Figure 5 FIG. is an example of a screen 30A of a display unit 30 showing a three-dimensional model 16B representing a panoramic view of a bridge 1. Figure 6 FIG. is an example of a screen 30A of the display unit 30 showing the three-dimensional model 16B of the bridge 1 that is stereoscopically moved by an operation such as zooming in.

[0080] In Figure 6 the bridge 1 shown on the screen 30A is composed of various components, including a main girder 2 erected between piers 7, a cross beam 3 provided in a direction orthogonal to the main girder 2 and connecting the main girders 2, and cross braces 4 and horizontal bracings 5 that connect the main girders 2 to resist lateral loads such as wind and earthquakes. A floor slab 6 for vehicles to travel on is poured on the upper part of the main girder 2, etc. The floor slab 6 is generally a reinforced concrete floor slab.

[0081] The floor slab 6 is usually based on a rectangular-shaped cofferdam divided by the main girder 2 and the cross beam 3, and when inspecting for damage (cracks, concrete peeling, etc.) of the floor slab 6, it is carried out in units of the cofferdam.

[0082] Therefore, each cofferdam of the bottom plate 6 is one of the management divisions of the inspection units of the parts and components of the structure (bridge). In addition, the parts / components divisions of the structure that can be the management divisions of the bridge, in addition to the bottom plate (cofferdam), also include parts / components divisions of the structure (main girder 2, cross beam 3, cross bracing frame 4, horizontal longitudinal bracing 5, bridge pier 7 (column part / wall part, beam part, corner / joint part)), etc.

[0083] The management information of each management division of the structure can be stored in the storage unit 16.

[0084] As the management information of each management division of the structure, orthophotos corresponding to the management divisions of the structure, orthophotos overlaid with damage information, damage maps, damage quantity tables, repair maps, or repair quantity tables, etc. can be considered. These management information are produced when the structure is inspected or repaired.

[0085] Figure 7 It is a figure showing an example of an orthophoto corresponding to the cofferdam of the bridge.

[0086] An orthophoto is an image obtained by orthogonally projecting the photographed image of the object (cofferdam) onto the surface of the cofferdam. The orthophoto of one cofferdam can be produced as follows: Extract a plurality of photographed images corresponding to the cofferdam from the group of photographed images stored in the storage unit 16, perform panoramic synthesis on the extracted plurality of photographed images, and perform projection conversion on the panoramically synthesized image so that the panoramically synthesized image becomes an image orthogonally projected onto the surface of the cofferdam.

[0087] The panoramic synthesis of a plurality of photographed images can be performed by an image processing that extracts a plurality of feature points in the overlapping areas between the mutually overlapping photographed images and makes the extracted plurality of feature points coincide respectively. In addition, the orthogonal projection of the panoramically synthesized image onto the surface of the cofferdam can be performed by a projection conversion that makes the positions corresponding to the four corners of the cofferdam in the panoramically synthesized image coincide with the three-dimensional positions of the four corners of the cofferdam.

[0088] Figure 8 It is a figure showing an example of a damage map corresponding to the cofferdam.

[0089] In Figure 8 In the shown damage map, five cracks C1 to C5 and the peeling H1 of the concrete are illustrated. The damage map can be generated by manually tracing the cracks, peeling and other damages visually recognized on the orthophoto, or performing image processing for automatically detecting damages based on the orthophoto and manually correcting as needed.

[0090] Figure 9 It is a figure showing an example of an orthophoto overlaid with a damage map corresponding to the cofferdam.

[0091] Figure 9 The orthophoto image overlaid with the damage map as shown can be obtained by Figure 7 overlaying the damage map as shown on Figure 8 the orthophoto image as shown.

[0092] The damage map is created by marking the damaged parts in a conspicuous color such as red. By overlaying the damage map on the orthophoto image, the damaged parts can be easily visually identified.

[0093] Figure 10 is a diagram showing an example of a damage quantity table corresponding to the cofferdam.

[0094] In Figure 10 the damage quantity table as shown, there are items such as damage ID (identification), damage type, dimensions (width), dimensions (length), dimensions (area), and information corresponding to each item is recorded for each damage.

[0095] Next, the image list 16C stored in the storage unit 16 will be described.

[0096] Figure 11 is a diagram showing an example of the image list 16C.

[0097] Figure 11 The image list 16C as shown contains the attributes of each photographic image in the photographic image group 16A, information for managing each photographic image, and information for associating each photographic image with the three-dimensional model 16B, etc.

[0098] Figure 11 The items shown in the image list 16C include the reduced image (thumbnail) of each photographic image, the identification information (image file name) for determining each photographic image, the alignment degree of the photographic image with respect to the structure, the distance of the photographic image from the structure, information indicating whether it is associated with the inspection record of the structure, information indicating whether it is associated with the damage detection results such as cracks (for example, Figure 8 the damage map as shown, Figure 10 the damage quantity table as shown, etc.), the position information (three-dimensional position information) of the structure captured in each photographic image, and the photographic date and time. In addition, in the image list 16C, not limited to the above examples, it may also include the sharpness of each photographic image, information indicating whether it is associated with the repair record (repair map), etc.

[0099] Here, in this example, the alignment degree of a photographic image refers to the angle formed between the normal direction of the surface of the structure corresponding to the photographic image and the photographic direction of the imaging device when the photographic image is taken. Therefore, the smaller the alignment degree (angle) of the photographic image, the more the image is taken directly facing the structure, and it is a good image. The distance between the photographic image and the structure is the distance between the structure corresponding to the photographic image and the imaging device when the photographic image is taken (photographic distance).

[0100] The position information of the structure photographed in the photographic image can be set as the representative three-dimensional points of the three-dimensional point group (for example, the three-dimensional points at the four corners of the photographic image or near them) in addition to the three-dimensional point group of the structure within the range photographed in the photographic image. The position information of the structure photographed in the photographic image becomes the information for associating the photographic image group 16A with the three-dimensional model 16B.

[0101] [Image display method]

[0102] <The first embodiment>

[0103] Figure 12 It is a flowchart showing the first embodiment of the image display method according to the present invention.

[0104] In Figure 12 while explaining the operations of the respective parts of the image display device 10 shown in Figure 1 and Figure 4 the first embodiment of the image display method according to the present invention will be described.

[0105] Figure 12 The image display method of the first embodiment shown in

[0106] is a method such that when a desired photographic image is selected from the photographic image group 16A, it is easy to confirm which position on the three-dimensional model 16B of the structure the selected photographic image corresponds to. Figure 12 In

[0107] As Figure 11 shown, the image list 16C includes the attributes of each photographic image in the photographic image group 16A, information for managing each photographic image, information for associating each photographic image with the three-dimensional model 16B, etc. The image list 16C can be created by the CPU 20 functioning as an image list creation unit 20D based on the photographic image group 16A and various information associated with the photographic image group 16A, but it can also be created by an external device and stored in the storage unit 16.

[0108] Next, the details of the process in step S100 will be described.

[0109] Figure 13 It is a diagram showing a subroutine representing a first example of the process in step S100.

[0110] In Figure 13 the CPU 20 that functions as the condition setting unit 30F sets the conditions (screening conditions) for screening the photographed images extracted from the photographed image group 16A by using the user operation of the operation unit 18 (step S101).

[0111] Here, among the screening conditions, there is information indicating whether the photographed image is associated with the inspection record of the structure, information indicating whether it is associated with the damage (crack) detection result, and information indicating whether it is associated with the repair record (repair drawing).

[0112] In addition, the screening conditions are not limited to the above examples. For example, information indicating the management division of the structure can be considered. As described above, the regular inspection of the structure is carried out according to the division of the inspection units of the parts and components constituting the structure, that is, for each management division. In the case where the structure is a bridge, the main girder, cross beam, bottom plate (cofferdam constituting the bottom plate), pier, etc. correspond to the management division. The information indicating these management divisions can be used as the screening conditions.

[0113] The CPU 20 that functions as the image list creation unit 20D screens the photographed image group 16A based on the screening conditions set by the condition setting unit 20F and creates an image list representing the screened photographed images (step S102).

[0114] The display control unit 26 that functions as the first display control unit causes the image list of the photographed images screened according to the screening conditions to be displayed on the display unit 30 (step S103, first display step). Thus, the image list displayed on the display unit 30 is updated according to the screening conditions set by the user.

[0115] For example, if the information indicating whether it is associated with the inspection record of the structure is set as the screening condition, an image list composed only of the photographed images associated with the inspection record can be displayed on the display unit 30. In addition, if the information indicating the management division of the structure is set as the screening condition, an image list composed only of the photographed images associated with the set management division (photographed images of the area of the structure corresponding to the set management division) can be displayed on the display unit 30.

[0116] Figure 14 It is a diagram showing a subroutine representing a second example of the process in step S100.

[0117] InFigure 14 In this case, the CPU 20 functioning as the condition setting unit 30F sets a condition (sorting condition) indicating the rearrangement of the photographed image group 16A by using the user operation of the operation unit 18 (step S104).

[0118] Here, the sorting conditions include the alignment degree of the photographed image with respect to the subject (structure), the distance of the photographed image from the structure, and the photographing date and time (see Figure 11 ).

[0119] In addition, the sorting conditions are not limited to the above examples. For example, other sorting conditions such as the sharpness of each photographed image may also be used. However, the image list 16C preferably has at least information corresponding to the sorting conditions.

[0120] The CPU 20 functioning as the image list creation unit 20D creates an image list obtained by rearranging the photographed image group 16A based on the sorting conditions set by the condition setting unit 20F (step S105).

[0121] The display control unit 26 functioning as the first display control unit causes the image list of the photographed image group 16A rearranged according to the sorting conditions to be displayed on the display unit 30 (step S106, first display step).

[0122] For example, the image list 16C (original image list) stored in the storage unit 16 is displayed on the display unit 30. By clicking on a desired item (sorting condition) in the displayed image list, the image list of the photographed image group 16A rearranged in ascending or descending order according to the sorting conditions can be displayed on the display unit 30.

[0123] Figure 15 is a diagram showing a third example of the process of step S100. In addition, in Figure 15 , for Figure 13 and Figure 14 The same step numbers are assigned to the parts common to the processes of the first and second examples shown, and their detailed descriptions are omitted.

[0124] Figure 15 The third example shown in Figure 13 and Figure 14 is an example combining the first and second examples shown in

[0125] First, the photographed images in the photographed image group 16A are screened based on the screening conditions (steps S101 and S102).

[0125] Next, an image list obtained by rearranging the screened photographed images based on the sorting conditions is created (steps S104 and S105), and the created image list (image list of the screened and rearranged photographed images) is displayed on the display unit 30 (step S107).

[0126] Return Figure 12 The CPU 20 determines whether to end the display of the image list on the display unit 30 (step S108). The CPU 20 determines whether there is an instruction input for ending the image display from the operation unit 18. When there is no instruction input for ending (in the case of "No"), it transfers to step S110. When there is an instruction input for ending (in the case of "Yes"), it ends the processing related to the image display.

[0127] In step S110, the CPU 20 determines whether a selection instruction for an arbitrary photographed image is received from the image list according to the user operation using the operation unit 18 that functions as the first operation unit.

[0128] When it is determined that the selection instruction is not received (in the case of "No"), it returns to step S100, continues to display the image list on the display unit 30, and becomes a state where a selection instruction for a photographed image can be received.

[0129] On the other hand, when it is determined that the selection instruction is received (in the case of "Yes"), the CPU 20 that functions as the position information acquisition unit 20E acquires the position information on the three-dimensional model 16B corresponding to the photographed image indicated by the selection instruction (step S120). The acquisition of this position information can be read from the image list 16C based on the image file name of the photographed image indicated by the selection instruction.

[0130] Next, the display control unit 26 that functions as the second display control unit switches the display on the display unit 30 from the display of the image list 16C to the display of the three-dimensional model 16B or the like. That is, the display control unit 26 reads out the three-dimensional model 16B stored in the storage unit 16 and displays it on the display unit 30, and based on the position information on the three-dimensional model 16B acquired by the position information acquisition unit 20E, overlaps and displays an identifier indicating the position of the photographed image indicated by the selection instruction on the three-dimensional model 16B displayed on the display unit 30 (step S130, second display step).

[0131] Figure 16 It is a diagram showing an example of the screen 30A of the display unit 30 that displays the photographed image 100 indicated by the selection instruction and the three-dimensional model 16B including the photographed range of the photographed image 100.

[0132] In Figure 16In the example shown, in order to easily confirm the position of the photographed image 100 on the three-dimensional model 16B, the three-dimensional model 16B is automatically enlarged, translated, and rotated to display a part of the three-dimensional model 16B. In addition, the view operations such as enlargement of the three-dimensional model 16B can be automatically performed based on the position information of the structure photographed in the photographed image 100 (for example, the three-dimensional positions of the four corners of the photographed image 100) recorded in the image list 16C.

[0133] In addition, in Figure 16 the example shown, the photographed image 100 and the three-dimensional model 16B are displayed side by side on the screen 30A of the display unit 30, and a mark 100A indicating the position of the photographed image 100 is overlapped and displayed on the three-dimensional model 16B. The mark 100A is displayed as a frame indicating the photographing range of the photographed image 100. The frame indicating the photographing range of the photographed image 100 may be formed by line segments connecting the three-dimensional positions of the four corners of the photographed image 100 respectively.

[0134] In addition, in Figure 16 the display of a part of the three-dimensional model 16B shown, when the position of the photographed image 100 in the whole three-dimensional model 16B cannot be grasped, the user can manually perform a view operation (shrinking) on the three-dimensional model 16B to display a large range of the three-dimensional model 16B (refer to Figure 6 ), or manually perform a view operation on the three-dimensional model 16B to display the panorama of the three-dimensional model 16B (refer to Figure 5 ).

[0135] When displaying a large range of the three-dimensional model 16B, as a mark indicating the position of the photographed image 100 overlapped and displayed on the three-dimensional model 16B, a high-brightness point or the flashing of a high-brightness point is preferred.

[0136] In addition, contrary to the above example, the panorama of the three-dimensional model 16B on which the mark indicating the position of the photographed image 100 was first overlapped and displayed (refer to Figure 5 ) is displayed on the display unit 30. In this case, it is preferred to automatically rotate the three-dimensional model 16B so that the mark can be confirmed. Thus, the user can generally grasp which part of the structure the photographed image 100 is an image of.

[0137] When the user wants to grasp in detail which part of the structure the photographed image 100 is an image of, by manually performing a view operation to enlarge the three-dimensional model 16B (a view operation to translate and / or rotate it as needed), the enlarged three-dimensional model 16B, that is, the three-dimensional model 16B on which the mark indicating the position of the photographed image 100 is overlapped and displayed, can be displayed on the display unit 30 (see Figure 6 , Figure 16)。

[0138] As described above, the user can easily grasp which position of the structure (3D model 16B) the photographic image indicated by the selection on the image list was taken from by the identifier indicating the position of the photographic image overlaid on the 3D model 16B.

[0139] Return Figure 12 , the CPU 20 functioning as the display switching unit 20C determines whether to switch the display of the display unit 30 from the display of the 3D model 16B or the like to the display of the image list 16C (step S140). The determination of switching to the image list 16C can be made based on a user operation using the operation unit 18. For example, when the user wants to select a different photographic image, the operation unit 18 can be used to perform a switching operation to switch to the display of the image list 16C.

[0140] In step S140, when switching from the 3D model 16B or the like to the display of the image list 16C (in the case of "Yes"), the CPU 20 proceeds to step S100.

[0141] Thereby, the image list 16C can be displayed on the display unit 30. When the CPU 20 functioning as the display switching unit 20C displays the image list 16C on the display unit 30, it can switch the function of the display control unit 26 from the second display control unit to the first display control unit.

[0142] On the other hand, in step S140, when it is determined not to switch to the display of the image list 16C (in the case of "No"), the CPU 20 proceeds to step S150.

[0143] In step S150, the CPU 20 determines whether there is an instruction input for ending the image display from the operation unit 18. If there is no instruction input for ending (in the case of "No"), it returns to step S130 and continues to display the 3D model 16B or the like on the display unit 30. On the other hand, if there is an instruction input for ending (in the case of "Yes"), the processing related to the image display is ended.

[0144] In this example, it is assumed that the display of the image list 16C and the display of the 3D model 16B or the like in the display unit 30 are switched based on a user operation or the like, but it can also be set to display the image list 16C and the 3D model 16B or the like simultaneously on the display unit 30.

[0145] <Second Embodiment>

[0146] Figure 17 It is a flowchart showing a second embodiment of the image display method according to the present invention.

[0147] In Figure 17 , while explaining the operations of each part of the image display device 10 shown in Figure 1 and Figure 4 , a second embodiment of the image display method according to the present invention will be described.

[0148] As Figure 2 shown, in the storage unit 16 of the image display device 10, a set of photographic images 16A obtained by photographing a structure (a bridge in this example), a three-dimensional model 16B, and an image list 16C are stored. In the case of implementing the image display method of this example, the storage unit 16 is prepared in advance.

[0149] In Figure 17 , the display control unit 26 that functions as the fourth display control unit first reads out the three-dimensional model 16B stored in the storage unit 16 and causes the three-dimensional model 16B to be displayed on the display unit 30 (step S210). Here, when the display control unit 26 first causes the three-dimensional model 16B to be displayed on the display unit 30, as Figure 5 shown, it is preferable to cause the three-dimensional model 16B representing the panoramic view of the bridge to be displayed on the screen 30A of the display unit 30 so that the entire bridge 1 as a structure can be grasped.

[0150] Next, the CPU 20 or the display control unit 26 determines whether the user has performed a view operation such as magnifying the three-dimensional model 16B displayed on the screen 30A (step S212). The view operation is an operation for magnifying, reducing, translating, or rotating the three-dimensional model 16B expressed in 3D (three dimensions) on the screen 30A, and is performed using the operation unit 18 that functions as the second operation unit. In this case, as the second operation unit for receiving the user's view operation, a 3D mouse is preferable, but an ordinary mouse or the like can also be used.

[0151] In step S212, when it is determined that a view operation has been performed (in the case of "Yes"), the CPU 20 or the display control unit 26 determines whether the view operation is an operation for magnifying or reducing the three-dimensional model 16B, an operation for translating the three-dimensional model 16B, or an operation for rotating it (steps S214, step S216).

[0152] When a view operation for magnifying or reducing the three-dimensional model 16B is performed through the operation unit 18, the display control unit 26, which functions as the fourth display control unit, creates display data for magnifying or reducing the three-dimensional model 16B according to the instruction for magnification or reduction based on the view operation, and outputs it to the display unit 30 (step S218). In addition, when a view operation for translating the three-dimensional model 16B is performed through the operation unit 18, the display control unit 26 creates display data for translating the three-dimensional model 16B according to the instruction for translation based on the view operation, and outputs it to the display unit 30 (step S220). When a view operation for rotationally moving the three-dimensional model 16B is performed through the operation unit 18, display data for rotationally moving the three-dimensional model 16B according to the instruction for rotational movement based on the view operation is created and output to the display unit 30 (step S222). The operations in steps S212 to S222 are repeated until all the requests in S212 are processed.

[0153] In step S212, when it is determined that no view operation has been performed (in the case of “No”), or when the processing in steps S218, S220, and S222 ends, the CPU 20 transfers to step S223.

[0154] In step S223, the CPU 20 determines whether an instruction input for ending the image display has been received from the operation unit 18. If no ending instruction input has been received (in the case of “No”), it transfers to step S224. If an ending instruction input has been received (in the case of “Yes”), the processing related to the image display is ended.

[0155] Next, the CPU 20 determines whether position information indicating a position on the three-dimensional model 16B displayed on the display unit 30 has been accepted according to a user operation using the operation unit 18, which functions as the third operation unit (step S224).

[0156] In the case where position information indicating a position on the three-dimensional model 16B has not been accepted (in the case of “No”), it transfers to step S210, and the display control unit 26 continues to display the three-dimensional model 16B on the display unit 30. In addition, in the case where a view operation is performed and display data for magnifying or reducing the three-dimensional model 16B, display data for translating the three-dimensional model 16B, or display data for rotationally moving the three-dimensional model 16B is created through steps S218, S220, or S222, the magnified, reduced, translated, or rotationally moved (stereoscopically moved) three-dimensional model 16B is displayed on the display unit 30 based on the latest display data.

[0157] Figure 6It is a diagram showing an example of the screen 30A of the display unit 30 that displays the three-dimensional model 16B of the bridge that is stereoscopically moved by view operations such as magnification.

[0158] That is, through the user's view operation, the three-dimensional model 16B of the bridge 1 displayed on the screen 30A of the display unit 30 can be changed from Figure 5 the three-dimensional model 16B showing the panoramic view of the bridge 1 as shown to the three-dimensional model 16B that is magnified, moved, and rotated as shown in Figure 6 In addition, a view operation is performed to spatially grasp the entire bridge with the three-dimensional model 16B while easily seeing the parts to be inspected.

[0159] In Figure 6 32 is a cursor indicating the input position in the screen 30A of the display unit 30, and the cursor 32 can be moved on the screen 30A by operating the operation unit 18 (pointing device such as a mouse).

[0160] When the user wants to confirm the desired inspection part of the bridge, while spatially grasping the entire bridge with the three-dimensional model 16B, the three-dimensional model 16B is stereoscopically moved to search for the desired inspection part on the screen 30A of the display unit 30. Then, on the screen 30A of the display unit 30, the cursor 32 is moved to the desired inspection part, and a mouse click operation or an input operation based on an execution key is performed. Thereby, the operation unit 18 can specify the position on the three-dimensional model 16B displayed on the screen 30A of the display unit 30 and can receive the position information indicating the specified position.

[0161] In Figure 6 the position of the cursor 32 is within the bottom plate 6 of the bridge 1. Here, preferably, when a mouse click operation or the like is performed, the display control unit 26 that functions as the fourth display control unit causes the mark 34 indicating the selection instruction of the desired inspection part within the bottom plate 6 to be overlapped and displayed at the position indicated by the cursor 32 on the three-dimensional model 16B.

[0162] Return to Figure 17 When it is determined in step S224 that the position information indicating the position on the three-dimensional model 16B magnified and displayed on the display unit 30 has been received (in the case of “Yes”), the process proceeds to step S226.

[0163] In step S226, the CPU 20 functioning as the photographed image search unit 20A first determines a three-dimensional position on the three-dimensional model 16B based on the position information of the position indicated by the cursor 32 on the screen 30A (the three-dimensional model 16B displayed thereon) of the display unit 30. The position information of the position indicated by the cursor 32 on the screen 30A of the display unit 30 can be obtained as coordinate information on the screen 30A. On the other hand, the three-dimensional model 16B displayed on the screen 30A of the display unit 30 is enlarged, translated, or rotated by a view operation.

[0164] Therefore, the position information (coordinate information) on the three-dimensional model 16B that can be displayed on the screen 30A of the display unit 30 by performing a view operation such as enlargement and the information on the magnification, translation amount, and rotation amount of the three-dimensional model 16B based on the view operation are used to determine the three-dimensional position on the three-dimensional model 16B.

[0165] Next, the CPU 20 functioning as the photographed image search unit 20A searches for a plurality of photographed images including pixels corresponding to the determined three-dimensional position from the group of photographed images 16A stored in the storage unit 16 (step S226).

[0166] When searching for a plurality of photographed images, information associating the group of photographed images 16A with the three-dimensional model 16B ( Figure 11 the position information of the structure photographed in the photographed image in the image list 16C shown) can be used. For example, in the image list 16C, by screening the group of photographed images 16A for photographed images having the same position information as or including the three-dimensional position on the determined three-dimensional model 16B, a plurality of photographed images can be searched. In addition, as described above, since each photographed image in the group of photographed images 16A has an overlapping repeated area, there are a plurality of photographed images in the group of photographed images 16A that include pixels corresponding to the three-dimensional position on the determined three-dimensional model 16B.

[0167] Next, the CPU 20 functioning as the photographed image determination unit 20B determines the best photographed image from the plurality of photographed images searched in step S226, or determines the priority order of the plurality of photographed images searched (step S227).

[0168] Here, the photographed image determination unit 20B can use the information registered in Figure 11Based on the alignment degree of the photographed image with respect to the subject (structure) or the distance of the photographed image with respect to the structure in the shown image list 16C, the best photographed image or the priority order is determined from multiple photographed images. For example, the photographed image determination unit 20B can determine the photographed image that is more correctly taken of the structure as the appropriate photographed image based on the alignment degree of each photographed image. Additionally, based on the distance of the photographed image with respect to the structure, the photographed image obtained by photographing the structure at a closer distance can be used as the appropriate photographed image. Furthermore, the distance of the photographed image with respect to the structure can use the photographing distance recorded as tag information in the image file (Exif (EXchangeable image file format) file) of the photographed image. Additionally, the information of the position of the imaging device (GPS (global positioning system) information) recorded as tag information in the Exif file can be used, and the photographing distance can be calculated based on the position of the imaging device and the three-dimensional position (position converted into GPS information) on the determined three-dimensional model 16B.

[0169] Furthermore, the photographed image determination unit 20B calculates the offset between the position on the photographed image corresponding to the determined three-dimensional position and the center position of the photographed image. The smaller the calculated offset, the more suitable the photographed image can be set. The higher the clarity (contrast) of the photographed image (including the local area of the pixels corresponding to the three-dimensional position on the determined three-dimensional model 16B), the more suitable the photographed image can be set.

[0170] The photographed image determination unit 20B determines the best photographed image from the multiple photographed images searched based on one or more of the above conditions such as alignment degree and distance, or determines the priority order of the multiple photographed images searched.

[0171] When the best photographed image or the priority order is determined by the photographed image determination unit 20B, the display control unit 26 that functions as the third display control unit reads out the determined best photographed image from the storage unit 16, causes the read best photographed image to be displayed on the display unit 30, or causes a part or all of the multiple photographed images to be displayed on the display unit 30 in accordance with the determined priority order (step S228).

[0172] Thereby, when indicating the position of the inspection point on the three-dimensional model 16B of the bridge that moves stereoscopically by view operation, the best photographed image or the priority order among the multiple photographed images searched based on the indicated position can be determined, and a part or all of the photographed images (hereinafter referred to as "best photographed images, etc.") are caused to be displayed on the display unit 30 in accordance with the determined best photographed image or priority order, and it is possible to easily confirm the photographed image of the desired inspection point.

[0173] Figure 18This is a diagram showing an example of a screen 30A of a display unit 30 that displays the optimal captured image 100 determined from multiple captured images.

[0174] In this example, when the CPU 20 functioning as the display switching unit 20C causes the optimal captured image 100 to be displayed on the display unit 30, the function of the display control unit 26 is switched from the fourth display control unit to the third display control unit. The display switching from the 3D model 16B to the optimal captured image 100 performed by the display switching unit 20C is determined by the captured image determination unit 20B, and can be automatically switched when the display of the optimal captured image 100 is possible.

[0175] Figure 19 This is a diagram showing another example of a screen 30A of a display unit 30 that displays the optimal captured image 100 determined from multiple captured images.

[0176] In Figure 19 In the example shown, one captured image 100 is displayed on the screen 30A of the display unit 30, and thumbnails 102 of multiple captured images are displayed at the lower part of the screen 30A. In addition, the thumbnails 102 of the multiple captured images are arranged in the priority order of the multiple captured images, and the optimal captured image 100 (the captured image with the highest priority) among the multiple captured images is displayed first.

[0177] By selecting a desired thumbnail 102 from the thumbnails 102 arranged in the priority order, the user can display the main image (captured image) corresponding to the thumbnail 102 on the screen 30A of the display unit 30.

[0178] Figure 20 This is a diagram showing yet another example of a screen 30A of a display unit 30 that displays the optimal captured image and the like determined from multiple captured images.

[0179] In Figure 20 In the example shown, multiple captured images 100 are displayed on the screen 30A of the display unit 30, and a scroll bar 36 is displayed at the lower part of the screen 30A. Priority orders are assigned to the multiple captured images, and the multiple captured images including the captured image 100 with the highest priority are displayed first.

[0180] When the user wants to view a captured image that is not displayed on the screen 30A of the display unit 30, the user can move (scroll) the captured images 100 displayed on the screen 30A of the display unit 30 by operating the scroll bar 36 using a mouse or the like, so as to view the desired captured image.

[0181] Return Figure 17, when the best photographic image or the like is displayed on the display unit 30, the CPU 20 then determines whether to switch the display on the display unit 30 from the best photographic image or the like to the three-dimensional model 16B (step S230). The determination of the switch from the best photographic image or the like to the three-dimensional model 16B can be made based on a user operation using the operation unit 18. For example, when the user wants to confirm photographic images of different inspection parts, the switch operation from the display of the best photographic image or the like to the display of the three-dimensional model 16B can be performed through the operation unit 18.

[0182] In step S230, when switching from the best photographic image or the like to the display of the three-dimensional model 16B (in the case of "Yes"), the CPU 20 transfers to step S210.

[0183] Thereby, the three-dimensional model 16B representing the panoramic view of the bridge can be displayed on the display unit 30 (refer to Figure 5 ). When the CPU 20 functioning as the display switching unit 20C causes the three-dimensional model 16B to be displayed on the display unit 30, it can switch the function of the display control unit 26 from the third display control unit to the fourth display control unit.

[0184] In addition, when switching from the best photographic image or the like to the display of the three-dimensional model 16B (in the case of "Yes"), the CPU 20 may also transfer to step S212. Thereby, it is possible to display the most recent three-dimensional model 16B when switching the display from the three-dimensional model 16B to the best photographic image or the like, which is suitable for the case where the user wants to confirm photographic images of inspection parts close to the previous inspection parts.

[0185] On the other hand, in step S230, when it is determined not to switch the display from the best photographic image or the like to the three-dimensional model 16B (in the case of "No"), it transfers to step S232.

[0186] In step S232, the CPU 20 determines whether there is an instruction input for ending the image display from the operation unit 18. In the case where there is no ending instruction input (in the case of "No"), it returns to step S228 and continues to cause the display unit 30 to display the best photographic image or the like. On the other hand, in the case where there is an ending instruction input (in the case of "Yes"), it ends the process related to the image display.

[0187] In this example, it is assumed that the display of the three-dimensional model 16B in the display unit 30 and the display of the best photographic image or the like are switched based on user operations or the like. However, it is not limited to this, and it can also be assumed that the fourth display control unit causes the three-dimensional model 16B to be displayed in the first display area of the display unit 30, and the third display control unit causes the best photographic image or the like to be displayed in the second display area of the display unit 30, and the two are displayed simultaneously.

[0188] However, since the three-dimensional model 16B in this example is a model in which a photographic image texture is mapped onto the polygonal faces of a polygon, by magnifying and displaying the three-dimensional model 16B, the characteristics of the surface of the structure can be confirmed to some extent, but small damages (for example, cracks with a width of 0.1 mm) cannot be confirmed. This is because the data volume of the three-dimensional model 16B is limited, and even if the three-dimensional model 16B is magnified, it cannot be visually recognized as the original photographic image or an image equivalent thereto.

[0189] In contrast, by using the three-dimensional model 16B of the structure, the desired inspection part can be easily determined, and by displaying the original photographic image (such as the optimal photographic image) corresponding to the determined inspection part, damages and the like of the inspection part can be easily confirmed.

[0190] [Other]

[0191] The three-dimensional model is not limited to being generated by using a set of photographic images obtained by photographing a subject and by the SfM method, and can be generated by various methods.

[0192] For example, two parallax images or three-dimensional information of a subject photographed by a dual-lens imaging device can be acquired, and the three-dimensional model of the structure can be generated by using the acquired three-dimensional information. In addition, a photographic image of a subject can be acquired by using a time-of-flight imaging device, and the three-dimensional coordinates of the subject corresponding to each pixel on the photographic image can be acquired to generate a three-dimensional model. Further, including a laser scanner having the function of an imaging device, the three-dimensional model of the structure can be generated based on the three-dimensional information of the structure acquired by the laser scanner.

[0193] The hardware of the image display device according to the present invention can be constituted by various processors. Among various processors, there are a general-purpose processor, namely a CPU (Central Processing Unit), which executes a program and functions as various processing units, a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit structure can be changed after being manufactured, and an application-specific circuit, such as an ASIC (Application Specific Integrated Circuit), which has a circuit structure specifically designed for executing specific processing. One processing unit of the image display device can be constituted by one of the above various processors, or can be constituted by two or more processors of the same or different types. For example, one processing unit can be constituted by a plurality of FPGAs or a combination of a CPU and an FPGA. In addition, a plurality of processing units can be constituted by one processor. As an example of constituting a plurality of processing units by one processor, first, there is a form in which, represented by a computer such as a client or a server, a combination of one or more CPUs and software constitutes one processor, and this processor functions as a plurality of processing units. Second, there is a form in which, represented by a system on chip (SoC) or the like, a processor that uses one IC (Integrated Circuit) chip to implement the functions of the entire system including a plurality of processing units is used. In this way, one or more of the above various processors are used as a hardware structure to constitute various processing units. Moreover, more specifically, the hardware structure of these various processors is a circuitry that combines circuit elements such as semiconductor elements.

[0194] In addition, the present invention includes an image display program and a storage medium recording the image display program. The image display program, when installed in a computer accessible to a storage unit, causes the computer to function as the image display device according to the present invention. The storage unit stores a three-dimensional model of a subject and a set of photographed images obtained by photographing the subject while changing the photographing position and the photographing direction with respect to the subject.

[0195] Moreover, it goes without saying that the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the present invention.

[0196] According to the above description, the image display device described in Supplementary Note Item 1 below and the image display method described in Supplementary Note Item 2 can be grasped.

[0197] [Supplementary Note Item 1]

[0198] An image display device, comprising:

[0199] A storage unit that stores a three-dimensional model of a subject and a set of photographed images obtained by changing a photographing position and a photographing direction with respect to the subject and photographing the subject;

[0200] A first display control processor that causes an image list representing the set of photographed images to be displayed on a display unit;

[0201] A first operation unit that receives a selection instruction of an arbitrary photographed image from the image list;

[0202] A position information acquisition processor that acquires position information on the three-dimensional model corresponding to the photographed image indicated by the selection instruction; and

[0203] A second display control processor that reads out the three-dimensional model stored in the storage unit and causes it to be displayed on the display unit, and the second display control processor causes an identifier indicating the position of the photographed image indicated by the selection instruction to be superimposed and displayed on the three-dimensional model displayed on the display unit based on the position information on the three-dimensional model acquired by the position information acquisition processor.

[0204] [Supplementary Note 2]

[0205] An image display method, comprising:

[0206] A step of preparing a storage unit that stores a three-dimensional model of a subject and a set of photographed images obtained by changing a photographing position and a photographing direction with respect to the subject and photographing the subject;

[0207] A first display step in which a first display control processor causes an image list representing the set of photographed images to be displayed on a display unit;

[0208] A step in which a first operation unit receives a selection instruction of an arbitrary photographed image from the image list;

[0209] A step in which a position information acquisition processor acquires position information on the three-dimensional model corresponding to the photographed image indicated by the selection instruction; and

[0210] A second display step in which a second display control processor reads out the three-dimensional model stored in the storage unit and causes it to be displayed on the display unit, and in the second display step, based on the acquired position information on the three-dimensional model, an identifier indicating the position of the photographed image indicated by the selection instruction is superimposed and displayed on the three-dimensional model displayed on the display unit.

[0211] Symbol Explanation

[0212] 1 Bridge

[0213] 2 Main Girder

[0214] 3 Crossbeam

[0215] 4 Cross brace

[0216] 5 Horizontal bracing

[0217] 6 Bed board

[0218] 7 Pier

[0219] 10 Image display device

[0220] 12 Image acquisition unit

[0221] 16 Storage unit

[0222] 16A Group of photographed images

[0223] 16B 3D model

[0224] 16C Image list

[0225] 18 Operation unit

[0226] 20 CPU

[0227] 20A Photographed image search unit

[0228] 20B Photographed image determination unit

[0229] 20C Display switching unit

[0230] 20D Image list creation unit

[0231] 20E Position information acquisition unit

[0232] 20F Condition setting unit

[0233] 22 RAM

[0234] 24 ROM

[0235] 26 Display control unit

[0236] 30 Display unit

[0237] 30A Screen

[0238] 30F Condition setting unit

[0239] 32 Cursor

[0240] 34, 100A Mark

[0241] 36 Scroll bar

[0242] 100 Photographed image

[0243] 102 Thumbnail

[0244] Steps S110 to S232

Claims

1. An image display device, comprising: a storage unit that stores a three-dimensional model of a subject and a set of photographed images obtained by photographing the subject while changing the photographing position and the photographing direction with respect to the subject; A first display control unit that causes an image list representing the set of photographed images to be displayed on a display unit; A first operation unit that receives an instruction to select an arbitrary photographed image from the image list; A position information acquisition unit that acquires position information on the three-dimensional model corresponding to the photographed image indicated by the selection instruction; And A second display control unit that reads out the three-dimensional model stored in the storage unit and causes it to be displayed on the display unit, and the second display control unit causes an identifier indicating the position of the photographed image indicated by the selection instruction to be superimposed and displayed on the three-dimensional model displayed on the display unit, based on the position information on the three-dimensional model acquired by the position information acquisition unit; The second display control unit causes the three-dimensional model on which the identifier is superimposed to be displayed in a first display area of the display unit, reads out the photographed image indicated by the selection instruction from the storage unit, and causes it to be displayed in a second display area of the display unit.

2. The image display device according to claim 1, wherein, The image display device includes: A condition setting unit that sets conditions indicating screening of photographed images extracted from the set of photographed images and / or rearrangement of photographed images; And An image list creation unit that creates the image list representing the photographed images screened from the set of photographed images and / or the image list after rearrangement of the photographed images, based on the conditions set by the condition setting unit; The first display control unit updates the image list displayed on the display unit using the image list created by the image list creation unit.

3. The image display device according to claim 1 or 2, wherein, Items displayed in the image list include one or more of a thumbnail image of each photographed image in the set of photographed images, identification information for identifying each photographed image, the alignment degree of the photographed image with respect to the subject, the distance of the photographed image from the subject, the sharpness of the photographed image, information indicating whether it is associated with an inspection record, damage detection result, or repair record of the subject, and the photographing date and time.

4. The image display device according to claim 1 or 2, wherein, The second display control unit enlarges, pans, or rotates the three-dimensional model displayed on the display unit based on the acquired position information on the three-dimensional model, so that the identifier superimposed on the three-dimensional model is easy to view.

5. The image display device according to claim 1 or 2, wherein, The image display device includes a second operation unit that receives a view operation of the three-dimensional model displayed on the display unit; The second display control unit causes the three-dimensional model to be displayed on the display unit based on the view operation received by the second operation unit.

6. The image display device according to claim 5, wherein, The view operation is an operation of enlarging, reducing, panning, or rotating the three-dimensional model displayed on the display unit; The second display control unit enlarges, reduces, pans, or rotates the three-dimensional model displayed on the display unit based on the view operation received by the second operation unit.

7. The image display device according to claim 1 or 2, wherein, The three-dimensional model is a model represented by a three-dimensional point group composed of three-dimensional information of a plurality of points on the surface of the subject, a model representing the surface of the subject by an aggregate of polygonal faces based on the three-dimensional point group, or a model obtained by texture-mapping a photographic image obtained by photographing the subject onto the polygonal faces.

8. An image display method, comprising: The step of preparing a storage unit that stores a three-dimensional model of a subject and a set of photographic images obtained by changing the photographic position and photographic direction with respect to the subject to photograph the subject; A first display control step in which a first display control unit causes an image list representing the set of photographic images to be displayed on a display unit; A step in which a first operation unit receives a selection instruction of an arbitrary photographic image from the image list; A step in which a position information acquisition unit acquires position information on the three-dimensional model corresponding to the photographic image indicated by the selection; And A second display control step in which a second display control unit reads out the three-dimensional model stored in the storage unit and causes it to be displayed on the display unit. In the second display control step, based on the acquired position information on the three-dimensional model, an identifier indicating the position of the photographic image indicated by the selection is superimposed and displayed on the three-dimensional model displayed on the display unit. In the second display control step, the three-dimensional model on which the identifier is superimposed is displayed in a first display area of the display unit, and the photographic image indicated by the selection is read out from the storage unit and displayed in a second display area of the display unit.

9. The image display method according to claim 8, wherein, The image display method includes: A step of setting, by a condition setting unit, conditions indicating screening of photographic images extracted from the set of photographic images and / or rearrangement of photographic images; and A step in which an image list production unit produces the image list representing the photographic images screened from the set of photographic images and / or the image list after rearrangement of the photographic images based on the set conditions, In the first display control step, the image list displayed on the display unit is updated using the produced image list.

10. The image display method according to claim 8 or 9, wherein, Items displayed in the image list include one or more of a reduced image of each photographic image in the set of photographic images, identification information for determining each photographic image, the alignment degree of the photographic image with respect to the subject, the distance of the photographic image from the subject, the sharpness of the photographic image, information indicating whether it is associated with an inspection record, a damage detection result, or a repair record of the subject, and the date and time of photography.

11. The image display method according to claim 8 or 9, wherein, In the second display control step, based on the acquired position information on the three-dimensional model, the three-dimensional model displayed on the display unit is enlarged, translated, or rotated so that the identifier superimposed on the three-dimensional model is easier to view.

12. The image display method according to claim 8 or 9, wherein, The image display method includes a step in which a second operation unit receives a view operation of the three-dimensional model displayed on the display unit, In the second display control step, the three-dimensional model is displayed based on the received view operation.

13. The image display method according to claim 12, wherein, The view operation is an operation of enlarging, reducing, translating, or rotating the three-dimensional model displayed on the display unit. In the second display step, based on the view operation, the three-dimensional model displayed on the display unit is enlarged, reduced, translated, or rotated.

14. The image display method according to claim 8 or 9, wherein, The three-dimensional model is a model represented by a three-dimensional point group composed of three-dimensional information of a plurality of points on the surface of the subject, a model representing the surface of the subject by an aggregate of polygonal faces based on the three-dimensional point group, or a model in which a photographic image obtained by photographing the subject is texture-mapped onto the polygonal faces.

15. A computer-readable recording medium storing an image display program, the image display program being installed in a computer accessible to a storage unit that stores a three-dimensional model of a subject and a set of photographic images obtained by photographing the subject while changing the photographing position and the photographing direction with respect to the subject, wherein, The image display program causes the computer to implement: a function of displaying an image list representing the set of photographic images on the display unit; a function of receiving a selection instruction of an arbitrary photographic image from the image list; a function of acquiring position information on the three-dimensional model corresponding to the photographic image indicated by the selection; a function of reading out the three-dimensional model stored in the storage unit and displaying it on the display unit, and in this function, based on the acquired position information on the three-dimensional model, an identifier indicating the position of the photographic image indicated by the selection is superimposed and displayed on the three-dimensional model displayed on the display unit; and a function of displaying the three-dimensional model on which the identifier is superimposed in a first display area of the display unit, and reading out and displaying the photographic image indicated by the selection from the storage unit in a second display area of the display unit.

Images