Image display device, method and program

Abstract

Provided are an image display device, a method, and a program that enable display of an optimal captured image or the like among a plurality of captured images including the pixels of desired three-dimensional points of a subject. By displaying a three-dimensional model of a subject on a display unit and performing a view operation, such as enlargement, on the three-dimensional model, a desired position on the subject can be easily designated. When a desired position on the subject is designated, the three-dimensional position on the three-dimensional model corresponding to the position is specified, and a plurality of captured images each including a pixel corresponding to the specified three-dimensional position are retrieved from a group of captured images of the subject. An optimal captured image is determined from among the plurality of retrieved captured images or a priority order is determined for the plurality of retrieved captured images, and the determined optimal captured image is displayed on the display unit, or some or all of the plurality of captured images are displayed on the display unit in accordance with the determined priority order.

Description

[Technical field]

[0001] The present invention relates to an image display device, method, and program, and more particularly to a technique for acquiring a desired photographed image from a group of photographed images taken by changing the photographing position and photographing direction of a subject. [Background technology]

[0002] Conventionally, an image management device has been proposed that can collectively acquire images of each structural unit of a structure from a large number of images of the structure (Patent Document 1).

[0003] The image management device described in Patent Document 1 displays a 3D point cloud related to a structure, or a 3D model consisting of a mesh generated based on the 3D point cloud, on a display unit, and when a user specifies the 3D coordinates of the structure using the displayed 3D point cloud or 3D model, extracts 3D points corresponding to segments, which are structural units in the structure, from the 3D point cloud based on the specified 3D coordinates. Here, a segment corresponds to a part of the structure to be inspected, such as a main girder or deck in the case of an inspection of a bridge.

[0004] This image management device obtains an image associated with a three-dimensional point corresponding to the extracted segment from an image storage unit that stores images of the structure, and displays the obtained image on a display unit. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2017-130146 A Summary of the Invention [Problem to be solved by the invention]

[0006] According to the image management device described in Patent Document 1, a segment, which is a structural unit of a structure (inspection target parts such as main girders and decks in the case of a bridge structure), is identified based on the three-dimensional coordinates of the structure specified by the user, and images associated with the segment are obtained and presented to the user. This allows the user to view images of the inspection target parts all at once, but the images presented by the image management device include images in which the position specified by the user (three-dimensional coordinates of the structure) is not photographed.

[0007] The image management device described in Patent Document 1 is designed to retrieve the desired image (image of the area to be inspected) from a large number of images of the structure when a user wishes to view all images of the area to be inspected of the structure at once, and present the image to the user.

[0008] That is, the image management device described in Patent Document 1 does not acquire and present an image in which a position specified by a user is photographed. Also, although a user specifies the three-dimensional coordinates of a structure using a three-dimensional point cloud or the like displayed on a display unit, this specification of the three-dimensional coordinates means specification of a part to be inspected, and does not specify an image including pixels of a three-dimensional point of the structure corresponding to the three-dimensional coordinates.

[0009] The present invention has been made in consideration of the above circumstances, and aims to provide an image display device, method, and program that can display the optimal captured image among multiple captured images that contain pixels of desired three-dimensional points of a subject such as a structure, or all or some of the multiple captured images in accordance with priority. [Means for solving the problem]

[0010] In order to achieve the above object, an image display device according to one embodiment of the present invention includes a memory unit that stores a three-dimensional model of a subject and a group of captured images of the subject by changing the shooting position and shooting direction with respect to the subject; a first operation unit that accepts a view operation of the three-dimensional model to be displayed on the display unit when the three-dimensional model stored in the memory unit is to be displayed on the display unit; a first display control unit that reads out the three-dimensional model stored in the memory unit and displays it on the display unit, and displays the three-dimensional model based on the view operation accepted by the first operation unit; a second operation unit that accepts position information indicating a position on the three-dimensional model displayed on the display unit in response to a user operation; a captured image search unit that identifies a three-dimensional position on the three-dimensional model based on the accepted position information and searches for a plurality of captured images including a pixel corresponding to the identified three-dimensional position from the group of captured images stored in the memory unit; a captured image determination unit that determines an optimal captured image from the searched plurality of captured images or determines a priority order of the searched plurality of captured images; and a second display control unit that causes the optimal captured image determined by the captured image determination unit to be displayed on the display unit, or causes some or all of the plurality of captured images to be displayed on the display unit according to the determined priority order.

[0011] According to one aspect of the present invention, a three-dimensional model of a subject is displayed on a display unit, and a desired position of the subject can be easily specified while looking down on the subject by performing view operations such as enlarging the three-dimensional model.

[0012] When a desired position of the subject is specified, the captured image search unit identifies a three-dimensional position on the three-dimensional model corresponding to the position, and searches a group of captured images of the subject for multiple captured images that include pixels corresponding to the identified three-dimensional position. Note that since each captured image of the group of captured images of the subject has an overlapping area, the group of captured images contains multiple captured images that include pixels that correspond to a certain three-dimensional position of the subject.

[0013] The captured image determination unit determines an optimal captured image from the multiple captured images retrieved, or determines a priority order for the multiple captured images retrieved, and the second display control unit causes the display unit to display the determined optimal captured image, or causes the display unit to display some or all of the multiple captured images in accordance with the determined priority order.

[0014] This makes it possible to display an optimal captured image among a plurality of captured images that includes a pixel at a desired three-dimensional point of the subject, or a part or all of the plurality of captured images, in accordance with the priority order.

[0015] In an image display device according to another aspect of the present invention, it is preferable that the captured image determination unit determines an optimal captured image or determines the priority order of the multiple captured images found based on one or more of the following conditions: the orientation of the captured image relative to the subject, the distance of the captured image relative to the subject, the amount of deviation between the position on the captured image corresponding to the identified three-dimensional position and the center position of the captured image, and the clarity of the captured image.

[0016] In an image display device according to yet another aspect of the present invention, the view operation is an operation of enlarging, reducing, translating, or rotating a three-dimensional model to be displayed on the display unit, and it is preferable that the first display control unit enlarges, reduces, translates, or rotates the three-dimensional model to be displayed on the display unit based on the view operation.

[0017] In an image display device according to yet another aspect of the present invention, the three-dimensional model is represented as a three-dimensional point cloud consisting of three-dimensional information about a large number of points on the surface of the subject, a representation of the surface of the subject as a collection of polygons based on the three-dimensional point cloud, or a representation in which an image of the subject is photographed and then texture-mapped onto the polygons.

[0018] In an image display device according to yet another aspect of the present invention, it is preferable that the first display control unit displays a three-dimensional model in a first display area of ​​the display unit, and the second display control unit displays a captured image in a second display area of ​​the display unit.

[0019] In an image display device according to still another aspect of the present invention, it is preferable to further include a display switching unit that switches between displaying the three-dimensional model by the first display control unit and displaying the captured image by the second display control unit on the display unit.

[0020] In yet another aspect of the image display device of the present invention, it is preferable that the image display device comprises a third display control unit that causes an image list indicating a group of captured images to be displayed on a display unit, a third operation unit that accepts an instruction to select an arbitrary captured image from the image list by user operation, a position information acquisition unit that acquires position information on a 3D model corresponding to the selected captured image, and a fourth display control unit that reads out the 3D model stored in the memory unit and displays it on the display unit, and which superimposes an indicator indicating the position of the selected captured image on the 3D model displayed on the display unit based on the position information on the 3D model acquired by the position information acquisition unit.

[0021] In yet another aspect of the image display device of the present invention, the device comprises a condition setting unit that sets conditions for narrowing down the captured images to be extracted from the group of captured images and / or for rearranging the group of captured images, and an image list creation unit that creates an image list indicating the captured images narrowed down from the group of captured images based on the conditions set by the condition setting unit and / or an image list in which the group of captured images has been rearranged, and it is preferable that the third display control unit updates the image list displayed on the display unit with the image list created by the image list creation unit.

[0022] In an image display device according to yet another aspect of the present invention, the items displayed in the image list preferably include one or more of a reduced image of each captured image in the group of captured images, identification information identifying each captured image, the orientation of the captured image relative to the subject, the distance of the captured image relative to the subject, the clarity of the captured image, the inspection record of the subject, information indicating whether it is linked to a damage detection result or repair record, and the date and time of capture.

[0023] In an image display device according to yet another aspect of the present invention, it is preferable that the fourth display control unit causes a three-dimensional model with an index superimposed thereon to be displayed in a first display area of ​​the display unit, and causes a selected captured image to be read out from the memory unit and displayed in a second display area of ​​the display unit.

[0024] In an image display device according to yet another aspect of the present invention, it is preferable that the fourth display control unit enlarges, translates, or rotates the three-dimensional model to be displayed on the display unit based on the acquired position information on the three-dimensional model, thereby making the indicators superimposed on the three-dimensional model easier to see.

[0025] An image display method according to yet another aspect of the present invention includes a step of preparing a storage unit that stores a three-dimensional model of a subject and a group of captured images of the subject captured by changing the shooting position and shooting direction with respect to the subject; a step of a first operation unit receiving a view operation of the three-dimensional model to be displayed on the display unit when the three-dimensional model stored in the storage unit is to be displayed on the display unit; a first display step of a first display control unit reading out the three-dimensional model stored in the storage unit and displaying it on the display unit, and displaying the three-dimensional model based on the view operation received by the first operation unit; and a second operation unit displaying the three-dimensional model displayed on the display unit in response to a user operation. the step of a captured image search unit identifying a three-dimensional position on the three-dimensional model based on the received position information and searching for a plurality of captured images including a pixel corresponding to the identified three-dimensional position from the group of captured images stored in the memory unit; the step of a captured image determination unit determining an optimal captured image from the plurality of retrieved captured images or determining a priority order for the plurality of retrieved captured images; and a second display step of a second display control unit causing the display unit to display the determined optimal captured image or causing the display unit to display some or all of the plurality of captured images in accordance with the determined priority order.

[0026] In an image display method according to yet another aspect of the present invention, the step of determining a captured image preferably determines an optimal captured image or determines a priority order for the retrieved captured images based on one or more conditions of the orientation of the captured image relative to the subject, the distance of the captured image relative to the subject, the amount of deviation between the position on the captured image corresponding to the identified three-dimensional position and the center position of the captured image, and the clarity of the captured image.

[0027] In an image display method according to yet another aspect of the present invention, the view operation is an operation of enlarging, reducing, translating, or rotating a three-dimensional model to be displayed on the display unit, and it is preferable that the first display step enlarges, reduces, translates, or rotates the three-dimensional model to be displayed on the display unit based on the view operation.

[0028] In an image display method according to yet another aspect of the present invention, the three-dimensional model is preferably represented as a three-dimensional point cloud consisting of three-dimensional information about a large number of points on the surface of the subject, a representation of the surface of the subject as a collection of polygons based on the three-dimensional point cloud, or a representation in which an image of the subject is photographed and then texture-mapped onto the polygons.

[0029] In an image display method according to yet another aspect of the present invention, it is preferable that the first display step displays a three-dimensional model in a first display area of ​​the display unit, and the second display step displays the captured image in a second display area of ​​the display unit.

[0030] In the image display method according to still another aspect of the present invention, it is preferable that the display switching section switches between displaying the three-dimensional model in the first display step and displaying the captured image in the second display step on the display section.

[0031] In yet another aspect of the present invention, it is preferable that the method includes a third display step in which a third display control unit causes the display unit to display an image list indicating a group of captured images; a step in which a third operation unit receives an instruction to select an arbitrary captured image from the image list by a user operation; a step in which a position information acquisition unit acquires position information on a three-dimensional model corresponding to the selected captured image; and a fourth display step in which a fourth display control unit reads out the three-dimensional model stored in the memory unit and displays it on the display unit, wherein the fourth display step superimposes an indicator indicating the position of the selected captured image on the three-dimensional model displayed on the display unit based on the acquired position information on the three-dimensional model.

[0032] In yet another aspect of the present invention, the image display method includes a step of setting conditions indicating narrowing down the captured images to be extracted from the group of captured images and / or rearranging the group of captured images by a condition setting unit, and a step of creating an image list indicating the captured images narrowed down from the group of captured images based on the set conditions and / or an image list in which the group of captured images has been rearranged, and it is preferable that the third display step updates the image list displayed on the display unit with the created image list.

[0033] In an image display method according to yet another aspect of the present invention, the items displayed in the image list preferably include one or more of a reduced image of each captured image in the group of captured images, identification information identifying each captured image, the orientation of the captured image relative to the subject, the distance of the captured image relative to the subject, the clarity of the captured image, inspection records for the subject, information indicating whether the image is linked to a damage detection result or repair record, and the date and time of capture.

[0034] In an image display method according to yet another aspect of the present invention, it is preferable that the fourth display step displays a three-dimensional model with an index superimposed thereon in a first display area of ​​the display unit, and reads out the selected captured image from the memory unit and displays it in a second display area of ​​the display unit.

[0035] In an image display method according to yet another aspect of the present invention, the fourth display step preferably enlarges, translates, or rotates the three-dimensional model to be displayed on the display unit based on the acquired position information on the three-dimensional model, thereby making the indicators superimposed on the three-dimensional model easier to see.

[0036] Yet another aspect of the invention is an image display program installed in a computer that can access a memory unit that stores a three-dimensional model of a subject and a group of captured images of the subject by changing the shooting position and shooting direction with respect to the subject, and causes the computer to realize the following functions when displaying the three-dimensional model stored in the memory unit on a display unit: accepting a view operation of the three-dimensional model; reading out the three-dimensional model stored in the memory unit and displaying it on the display unit, and displaying the three-dimensional model based on the view operation; accepting position information indicating a position on the three-dimensional model displayed on the display unit in response to a user operation; identifying a three-dimensional position on the three-dimensional model based on the accepted position information, and searching for multiple captured images including a pixel corresponding to the identified three-dimensional position from the group of captured images stored in the memory unit; determining an optimal captured image from the searched multiple captured images, or determining a priority order of the searched multiple captured images; and displaying the determined optimal captured image on the display unit, or displaying some or all of the multiple captured images on the display unit according to the determined priority order.

[0037] In an image display program relating to yet another aspect of the present invention, it is preferable for a computer to realize the following functions: displaying an image list indicating a group of captured images on a display unit; receiving a selection instruction for a captured image selected from the image list by user operation; acquiring position information on a three-dimensional model corresponding to the selected captured image; and reading out the three-dimensional model stored in a memory unit and displaying it on the display unit, wherein, based on the acquired position information on the three-dimensional model, an indicator indicating the position of the selected captured image is superimposed on the three-dimensional model displayed on the display unit. An image display device according to yet another aspect of the present invention includes a memory unit that stores a three-dimensional model of a subject and a group of captured images of the subject by changing the shooting position and shooting direction with respect to the subject, and a processor that, when displaying the three-dimensional model stored in the memory unit on a display unit, receives a view operation of the three-dimensional model, reads out the three-dimensional model stored in the memory unit and displays it on the display unit, displays the three-dimensional model based on the view operation, receives position information indicating a position on the three-dimensional model displayed on the display unit in response to a user operation, identifies a three-dimensional position on the three-dimensional model based on the received position information, searches for a plurality of captured images including a pixel corresponding to the identified three-dimensional position from the group of captured images stored in the memory unit, determines an optimal captured image from the plurality of captured images searched for or determines a priority order of the plurality of captured images searched for, and displays the determined optimal captured image on the display unit, or displays some or all of the plurality of captured images on the display unit according to the determined priority order. Effect of the Invention

[0038] According to the present invention, it is possible to search for multiple images taken at a position specified by a user from a group of images taken of a subject such as a structure, and to display the most suitable image from among the multiple images, or some or all of the multiple images, in order of priority. [Brief description of the drawings]

[0039] [Figure 1] FIG. 1 is a block diagram showing an example of a hardware configuration of an image display device according to the present invention. [Diagram 2] FIG. 2 is a conceptual diagram showing a group of captured images, a three-dimensional model, and an image list stored in the storage unit. [Diagram 3] FIG. 3 is a conceptual diagram showing how a three-dimensional model and an image list are generated based on a group of captured images. [Figure 4] FIG. 4 is a functional block diagram showing each function of the CPU. [Diagram 5] FIG. 5 is a diagram showing an example of a screen of a display unit on which a three-dimensional model showing a full view of a bridge, which is a subject, is displayed. [Figure 6] FIG. 6 is a diagram showing an example of a screen of the display unit displaying a three-dimensional model of a bridge that has been moved three-dimensionally by a view operation such as enlargement. [Figure 7] FIG. 7 is a diagram showing an example of an ortho-image corresponding to the deck space, which is one of the management categories of a bridge. [Figure 8] FIG. 8 is a diagram showing an example of a damage diagram corresponding to a coffer. [Figure 9] FIG. 9 is a diagram showing an example of an orthoimage on which a damage diagram corresponding to a coffer is superimposed. [Figure 10] FIG. 10 is a chart showing an example of a damage quantity table corresponding to the lattice. [Figure 11] FIG. 11 is a diagram showing an example of an image list. [Figure 12] FIG. 12 is a flowchart showing a first embodiment of an image display method according to the present invention. [Figure 13] FIG. 13 is a diagram showing an example of a screen of a display unit on which an optimal captured image determined from a plurality of captured images is displayed. [Figure 14] FIG. 14 is a diagram showing another example of the screen of the display unit on which the optimal captured image determined from among a plurality of captured images is displayed. [Figure 15] FIG. 15 is a diagram showing yet another example of the screen of the display unit on which the optimal captured image etc. determined from among a plurality of captured images is displayed. [Figure 16] FIG. 16 is a flowchart showing a second embodiment of the image display method according to the present invention. [Figure 17] FIG. 17 is a diagram showing a subroutine illustrating a first example of the process of step S100 shown in FIG. [Figure 18] FIG. 18 is a diagram showing a subroutine illustrating a second example of the process of step S100 shown in FIG. [Figure 19] FIG. 19 is a diagram showing a subroutine illustrating a third example of the process of step S100 shown in FIG. [Figure 20]FIG. 20 is a diagram showing an example of a screen of the display unit on which a selectively instructed photographed image and a three-dimensional model including the photographing range of the photographed image are displayed. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0040] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of an image display device, method, and program according to the present invention will be described with reference to the accompanying drawings.

[0041] [Hardware configuration of image display device] FIG. 1 is a block diagram showing an example of a hardware configuration of an image display device according to the present invention.

[0042] A personal computer or a workstation can be used as the image display device 10 shown in Fig. 1. The image display device 10 of this example is mainly composed of an image acquisition unit 12, a storage unit 16, an operation unit 18, a CPU (Central Processing Unit) 20, a RAM (Random Access Memory) 22, a ROM (Read Only Memory) 24, and a display control unit 26.

[0043] The image acquisition unit 12 corresponds to an input / output interface, and acquires images of a subject, etc. The subject in this example is a structure to be inspected, such as a bridge or a tunnel.

[0044] The images acquired by the image acquisition unit 12 are, for example, a large number of images (a group of captured images) of a subject (a structure) captured by a drone (an unmanned aerial vehicle) equipped with a camera. It is preferable that the group of captured images covers the entire structure, and that adjacent captured images overlap to a range of about 80%. Therefore, although it depends on the size of the structure, the group of captured images will be 1000 or more.

[0045] The group of captured images acquired by the image acquisition unit 12 is stored in the storage unit 16.

[0046] The storage unit 16 is a storage unit configured from a hard disk device, a flash memory, etc., and as shown in FIG. 2, the storage unit 16 stores a captured image group 16A, a three-dimensional model of a structure 16B, and an image list 16C.

[0047] The three-dimensional model 16B and the image list 16C can be generated based on the captured image group 16A as shown in the conceptual diagram of Fig. 3. The three-dimensional model 16B and the image list 16C will be described in detail later.

[0048] The storage unit 16 also stores an operating system, an image display program according to the present invention, various parameters, and the like.

[0049] The operation unit 18 includes a UI (User Interface) such as a keyboard and a mouse connected to the computer by wire or wirelessly. In this example, the keyboard and mouse function as an operation unit that accepts normal operation inputs of a computer, as well as a first operation unit that accepts view operations including enlarging a three-dimensional model to be displayed on the screen of the display unit 30, a second operation unit that accepts position information indicating a position on the three-dimensional model displayed on the display unit 30 in response to a user operation, and a third operation unit that accepts an instruction to select an arbitrary captured image from an image list by a user operation.

[0050] The operation unit 18 also functions as an operation unit for a condition setting unit that sets conditions (narrowing conditions) for narrowing down captured images to be extracted from the captured image group and / or conditions (sorting conditions) for rearranging the captured image group.

[0051] The CPU 20 receives input from the operation unit 18 and performs various processes. Specifically, the CPU 20 reads out various programs stored in the storage unit 16 or the ROM 24, etc., and controls each unit based on the input from the operation unit 18. By executing the image display program according to the present invention, the CPU 20 also has functions of a captured image search unit 20A, a captured image determination unit 20B, a display switching unit 20C, an image list creation unit 20D, a position information acquisition unit 20E, a condition setting unit 20F, etc., as shown in Fig. 4.

[0052] The RAM 22 is used as a working area for the CPU 20, and is used as a storage unit for temporarily storing read programs and various data.

[0053] The display control unit 26 is composed of a GPU (Graphics Processing Unit) and the like, and creates display data to be displayed on the display unit 30. The display control unit 26 is a part that outputs to the display unit 30, and in this example, functions as a first display control unit that reads out a three-dimensional model stored in the storage unit 16, displays the three-dimensional model on the display unit 30, and enlarges and displays the three-dimensional model based on a view operation including enlarging the three-dimensional model by the operation unit 18, and also functions as a second display control unit that reads out a captured image determined by the captured image determination unit 20B (CPU 20) from the storage unit 16 and displays the read captured image on the display unit 30. The display control unit 26 may be realized by the CPU 20 instead of the above-mentioned GPU.

[0054] Display control unit 26 also functions as a third display control unit that displays the image list created by image list creation unit 20D (CPU 20) on display unit 30. Display control unit 26 further functions as a fourth display control unit that reads out a three-dimensional model stored in storage unit 16 and displays it on display unit 30, and also functions as a fourth display control unit that superimposes and displays an indicator indicating the position of a captured image selected and instructed by operation unit 18 (third operation unit) on the three-dimensional model displayed on display unit 30, based on position information on the three-dimensional model acquired by position information acquisition unit 20E.

[0055] The display unit 30 uses various monitors such as an LCD monitor that can be connected to a computer, and displays various information such as a 3D model, captured images, and image lists based on display data input from the display control unit 26. It is also used as part of the user interface together with the operation unit 18.

[0056] In the image display device 10 having the above configuration, the CPU 20 reads out the image display program of the present invention stored in the memory unit 16 or the ROM 24 in response to an instruction input from the operation unit 18, and executes the image display program, thereby causing the display unit 30 to display various types of information in accordance with operations performed by the operation unit 18.

[0057] The 3D model includes a 3D point cloud that is generated by extracting feature points between overlapping images of a structure, estimating the position and attitude of the camera mounted on the drone based on the extracted feature points, and simultaneously estimating the 3D positions of the feature points from the estimated results of the camera's position and attitude.

[0058] There is a Structure from Motion (SfM) method that tracks the movement of multiple feature points in a group of images captured by a drone as the camera's shooting position moves, and simultaneously estimates the 3D structure of a structure (Structure) and the camera's posture (Motion). In recent years, an optimization calculation method called bundle adjustment has been developed, which makes it possible to produce highly accurate output.

[0059] It should be noted that the camera parameters (focal length, image size of the image sensor, pixel pitch, etc.) required when applying the SfM technique can be those stored in the storage unit 16. In addition, since the SfM technique cannot obtain an absolute scale, for example, the absolute scale (three-dimensional position) can be obtained by specifying a known size of a structure (distance between two points, etc.).

[0060] Here, the three-dimensional model may be one that is represented by a three-dimensional point cloud of a large number of points on the surface of the structure, one that represents the surface of the structure as a collection of polygonal polygons (e.g., triangular patches) based on the three-dimensional point cloud, or one that is texture-mapped with a photographed image (texture) of the structure onto a polygonal polygon. The three-dimensional model of the structure in this example is one that is texture-mapped with a photographed image onto a polygonal polygon.

[0061] Meanwhile, periodic inspections of a structure are carried out for each management section, which is a section of the inspection unit for the parts and members that make up the structure.

[0062] The following explanation will be given using a bridge as an example of a structure.

[0063] FIG. 5 is a diagram showing an example of a screen 30A of the display unit 30 on which a three-dimensional model 16B showing a full view of the bridge 1 is displayed, and FIG. 6 is a diagram showing an example of a screen 30A of the display unit 30 on which a three-dimensional model 16B of the bridge 1 is displayed that has been moved three-dimensionally by a view operation such as enlargement.

[0064] In Fig. 6, the bridge 1 displayed on screen 30A is composed of various components including main girders 2 spanning between piers 7, cross beams 3 that are installed perpendicular to the main girders 2 and connect the main girders together, and cross beams 4 and 5 that connect the main girders 2 to each other to resist lateral loads such as wind and earthquakes. A deck 6 for vehicles to run on is poured above the main girders. The deck 6 is generally made of reinforced concrete.

[0065] The deck 6 typically has a rectangular space defined by the main girders 2 and cross girders 3 as its basic unit, and inspection of the deck for damage (cracks, concrete spalling, etc.) is carried out on a space-by-space basis.

[0066] Therefore, each lattice of the deck is one of the inspection unit control categories for the parts and components that make up the structure (bridge). In addition to the deck (lattice), the parts and components that make up the structure that can be the control category for a bridge include the parts and components that make up the structure (main girder 2, cross beam 3, diagonal brace 4, cross beam 5, pier 7 (column, wall, beam, corner, joint)).

[0067] The storage unit 16 can store management information for each management classification of the structure.

[0068] Possible management information for each management category of a structure includes an orthoimage corresponding to the management category of the structure, an orthoimage with damage information superimposed, a damage map, a damage quantity list, a repair map, a repair quantity list, etc. This management information is created when the structure is inspected or repaired.

[0069] FIG. 7 is a diagram showing an example of an ortho-image corresponding to the space between beams of a bridge.

[0070] An orthoimage is an image in which a photographed image of a subject (a room) is orthogonally projected onto the surface of the room. An orthoimage of one room can be created by extracting a plurality of photographed images corresponding to the room from the photographed images stored in the storage unit 16, panorama-combining the extracted plurality of photographed images, and performing projective transformation on the panorama-combined image so that the panorama-combined image is orthogonally projected onto the surface of the room.

[0071] The panoramic synthesis of a plurality of captured images can be performed by extracting a plurality of feature points of overlapping areas between the captured images that overlap each other, and performing image processing to match the extracted plurality of feature points. In addition, the orthogonal projection of the panoramic synthesis image onto the surface of the partition can be performed by projective transformation to match the positions corresponding to the four corners of the partition in the panoramic synthesis image with the three-dimensional positions of the four corners of the partition.

[0072] FIG. 8 is a diagram showing an example of a damage diagram corresponding to a coffer.

[0073] The damage map shown in Figure 8 illustrates five cracks C1 to C5 and concrete spalling H1. The damage map can be generated by manually tracing damage such as cracks and spalling visually recognized on the orthoimage, or by performing image processing to automatically detect damage from the orthoimage and then manually correcting it as necessary.

[0074] FIG. 9 is a diagram showing an example of an orthoimage on which a damage diagram corresponding to a coffer is superimposed.

[0075] The orthoimage with the damage diagram superimposed thereon shown in FIG. 9 can be created by superimposing the damage diagram shown in FIG. 8 on the orthoimage shown in FIG.

[0076] The damage map is created by marking damaged areas with a conspicuous color such as red, and by overlaying the damage map on the orthoimage, the damaged areas can be easily visually identified.

[0077] FIG. 10 is a chart showing an example of a damage quantity table corresponding to the lattice.

[0078] The damage quantity table shown in FIG. 10 has items such as damage ID (identification), damage type, size (width), size (length), and size (area), and information corresponding to each item is recorded for each damage.

[0079] Next, the image list 14C stored in the storage unit 16 will be described.

[0080] FIG. 11 is a diagram showing an example of the image list 16C.

[0081] The image list 16C shown in FIG. 11 includes attributes of each captured image in the captured image group 16A, information for managing each captured image, information for linking each captured image with the three-dimensional model 16B, and the like.

[0082] Items displayed in image list 16C shown in Fig. 11 include reduced images (thumbnails) of each captured image, identification information (image file name) for identifying each captured image, the orientation of the captured image relative to the structure, the distance of the captured image relative to the structure, information indicating whether it is linked to the inspection record of the structure, information indicating whether it is linked to damage detection results such as cracks (for example, the damage diagram shown in Fig. 8 and the damage quantity table shown in Fig. 9), position information (three-dimensional position information) of the structure shown in each captured image, and the capture date and time. Note that image list 16C is not limited to the above examples, and can also include information indicating whether it is linked to a repair record (repair diagram), etc.

[0083] Here, the degree of facing of the captured image refers to the angle between the normal direction of the surface of the structure corresponding to the captured image and the shooting direction of the camera when the captured image was captured in this example. Therefore, the smaller the degree of facing (angle) of the captured image, the more directly the image is captured relative to the structure, and the better the image. The distance of the captured image from the structure is the distance (shooting distance) between the structure corresponding to the captured image and the camera when the captured image was captured.

[0084] The position information of the structure shown in the photographed image can be a 3D point group of the structure within the range shown in the photographed image, as well as a representative 3D point of the 3D point group (for example, the four corners of the photographed image or a 3D point in the vicinity thereof). This position information of the structure shown in the photographed image becomes information that links the photographed image group 16A and the 3D model 16B.

[0085] [Image display method] First Embodiment FIG. 12 is a flowchart showing a first embodiment of an image display method according to the present invention.

[0086] In FIG. 12, the operation of each part of the image display device 10 shown in FIGS. 1 and 4 will be described, while the first embodiment of the image display method according to the present invention will be described.

[0087] 2, a group of captured images 16A of a structure (a bridge in this example), a three-dimensional model 16B, and an image list 16C are stored in the storage unit 16 of the image display device 10. When the image display method of this example is carried out, such storage unit 16 is prepared in advance.

[0088] 12, the display control unit 26 functioning as the first display control unit first reads out the three-dimensional model 16B stored in the storage unit 16, and displays the three-dimensional model 18B on the display unit 30 (step S10, first display step). Here, when the display control unit 26 first displays the three-dimensional model 16B on the display unit 30, it is preferable to display the three-dimensional model 16B showing the entire bridge, which is a structure, on the screen 30A of the display unit 30 so that the entire bridge 1, which is a structure, can be grasped, as shown in FIG.

[0089] Next, the CPU 20 or the display control unit 26 determines whether or not a view operation such as enlargement has been performed by the user on the three-dimensional model 16B displayed on the screen 30A (step S12). The view operation is an operation for enlarging, reducing, translating, or rotating the three-dimensional model 16B expressed in 3D (three dimensions) on the screen 30A, and is performed using the operation unit 18 functioning as a first operation unit. In this case, a 3D mouse is suitable as the first operation unit that accepts the user's view operation, but a normal mouse or the like can also be used.

[0090] In step S12, when it is determined that a view operation has been performed (if "Yes"), the CPU 20 or the display control unit 26 determines whether the view operation is an operation to enlarge or reduce the three-dimensional model 16B, an operation to translate the three-dimensional model 16B, or an operation to rotate the three-dimensional model 16B (steps S14, S16).

[0091] When a viewing operation for enlarging or reducing the three-dimensional model 16B is performed by the operation unit 18, the display control unit 26 functioning as the first display control unit creates display data for enlarging or reducing the three-dimensional model 16B in response to the instruction for enlarging or reducing by the viewing operation, and outputs the display data to the display unit 30 (step S18). When a viewing operation for translating the three-dimensional model 16B is performed by the operation unit 18, the display control unit 26 creates display data for translating the three-dimensional model 16B in response to the instruction for translation by the viewing operation, and outputs the display data to the display unit 30 (step S20). When a viewing operation for rotating the three-dimensional model 16B is performed by the operation unit 18, the display control unit 26 creates display data for rotating the three-dimensional model 16B in response to the instruction for rotation by the viewing operation, and outputs the display data to the display unit 30 (step S22).

[0092] In step S12, when it is determined that a view operation has not been performed (in the case of "No"), or when the processes of steps S18, S20, and S22 are completed, the CPU 20 transitions to step S23.

[0093] In step S23, the CPU 20 determines whether or not an instruction to end the image display has been input from the operation unit 18, and if there is no instruction to end the image display (if "No"), it transitions to step S24, and if there is an instruction to end the image display (if "Yes"), it terminates the processing related to the image display.

[0094] Next, the CPU 20 determines whether or not position information indicating a position on the three-dimensional model 16B enlarged and displayed on the display unit 30 has been accepted in response to a user operation on the operation unit 18 functioning as a second operation unit (step S24).

[0095] If the position information indicating the position on the three-dimensional model 16B has not been received (if "No"), the process proceeds to step S10, and the display control unit 26 continues to display the three-dimensional model 16B on the display unit 30. When a view operation is performed and display data for enlarging or reducing the three-dimensional model 16B, display data for translating the three-dimensional model 16B, or display data for rotating the three-dimensional model 16B is created in step S18, S20, or S22, the three-dimensional model 16B that has been enlarged or reduced, translated, or rotated (moved three-dimensionally) based on the latest display data is displayed on the display unit 30.

[0096] FIG. 6 is a diagram showing an example of a screen 30A of the display unit 30, which displays a three-dimensional model 16B of a bridge that has been moved three-dimensionally by a view operation such as enlargement.

[0097] That is, by the user's view operation, the 3D model 16B of the bridge 1 displayed on the screen 30A of the display unit 30 can be transitioned from the 3D model 16B showing the entire bridge 1 shown in Fig. 5 to the 3D model 16B that has been enlarged, moved and rotated as shown in Fig. 6. The view operation is performed to make it easier to see the points requiring inspection while grasping the spatial understanding of the entire bridge on the 3D model 16B.

[0098] In FIG. 6, reference numeral 32 denotes a cursor that indicates an input position on a screen 30A of the display unit 30, and the cursor 32 can be moved on the screen 30A by operating the operation unit 18 (a pointing device such as a mouse).

[0099] When a user wishes to check a desired inspection location on a bridge, the user searches for the desired inspection location on the screen 30A of the display unit 30 by moving the 3D model 16B three-dimensionally while spatially grasping the entire bridge on the 3D model 16B. Then, the user moves the cursor 32 to the desired inspection location on the screen 30A of the display unit 30, and performs a click operation with the mouse or an input operation with the execute key. This enables the operation unit 18 to specify a position on the 3D model 16B displayed on the screen 30A of the display unit 30, and to receive position information indicating the specified position.

[0100] 6, cursor 32 is located within the deck 6 of bridge 1. When a mouse click or the like is performed, display control unit 26 functioning as the first display control unit preferably displays an indicator 34, which indicates that an instruction to select a desired inspection location within deck 6 has been given, superimposed on the position indicated by cursor 32 on 3D model 16B.

[0101] Returning to FIG. 12, if it is determined in step S24 that position information indicating a position on the three-dimensional model 16B enlarged and displayed on the display unit 30 has been accepted (if "Yes"), the process proceeds to step S26.

[0102] In step S26, the CPU 20 functioning as the captured image search unit 20A first identifies a three-dimensional position on the three-dimensional model 16B based on position information of the position indicated by the cursor 32 on the screen 30A of the display unit 30 (the three-dimensional model 16B displayed thereon). The position information of the position indicated by the cursor 32 on the screen 30A of the display unit 30 can be acquired as coordinate information on the screen 30A. Meanwhile, the three-dimensional model 16B displayed on the screen 30A of the display unit 30 can be enlarged, translated, and rotated by a view operation.

[0103] Therefore, the three-dimensional position on the three-dimensional model 16B can be identified based on position information (coordinate information) on the three-dimensional model 16B displayed on the screen 30A of the display unit 30 after a view operation such as enlargement has been performed, and information on the magnification ratio, parallel movement amount, and rotational movement amount of the three-dimensional model 16B due to the view operation.

[0104] Next, the CPU 20 functioning as the captured image search unit 20A searches for multiple captured images that contain pixels corresponding to the identified three-dimensional position from the captured image group 16A stored in the memory unit 16 based on the identified three-dimensional position on the three-dimensional model 16B (step S27).

[0105] When searching for a plurality of captured images, information linking the captured image group 16A and the three-dimensional model 16B (position information of a structure shown in the captured image in the image list 16C shown in FIG. 11) can be used. For example, a plurality of captured images can be searched for by narrowing down the captured image group 16A to a captured image having position information that is the same as the three-dimensional position on the identified three-dimensional model 16B or includes the three-dimensional position on the identified three-dimensional model 16B in the image list 16C. As described above, the captured image group 16A includes an overlapping area in which each captured image of the captured image group 16A overlaps with each other, and therefore, the captured image group 16A includes a plurality of captured images including a pixel corresponding to the three-dimensional position on the identified three-dimensional model 16B.

[0106] Next, the CPU 20 functioning as the photographed image determination section 20B determines an optimal photographed image from the plurality of photographed images retrieved in step S27, or determines the priority order of the plurality of photographed images retrieved (step S28).

[0107] Here, the captured image determination unit 20B can determine the optimal captured image or priority order from the multiple captured images using the facing degree of the captured image with respect to the subject (structure) or the distance of the captured image with respect to the structure registered in the image list 16C shown in FIG. 11. For example, the captured image determination unit 20B can determine the captured image captured with the structure more facing forward as the preferred captured image based on the facing degree of each captured image. Also, the captured image captured at a closer distance to the structure can be determined as the preferred captured image based on the distance of the captured image with respect to the structure. Note that the distance of the captured image with respect to the structure can be the shooting distance recorded as tag information of the image file (Exchangeable image file format (Exif) file) of the captured image. Also, the shooting distance can be calculated from the camera position and the identified three-dimensional position on the three-dimensional model 16B (position converted into GPS information) using the camera position information (GPS (global positioning system) information) recorded as tag information of the Exif file.

[0108] Furthermore, the captured image determination unit 20B calculates the amount of deviation between the position on the captured image corresponding to the specified three-dimensional position and the center position of the captured image, and the smaller the calculated amount of deviation, the more suitable the captured image can be. Also, the higher the clarity (contrast) of the captured image (local area including pixels corresponding to the specified three-dimensional position on the three-dimensional model 16B), the more suitable the captured image can be.

[0109] The captured image determination unit 20B determines the most appropriate captured image from among the retrieved captured images based on one or more of the above-mentioned conditions such as the facing degree and distance, or determines the priority order of the retrieved captured images.

[0110] When the optimal captured image or priority order is determined by the captured image determination unit 20B, the display control unit 26, functioning as a second display control unit, reads out the determined optimal captured image from the memory unit 16 and displays the read out optimal captured image on the display unit 30, or displays some or all of the multiple captured images on the display unit 30 in accordance with the determined priority order (step S28, second display step).

[0111] As a result, when the position of a location requiring inspection is specified on the 3D model 16B of the bridge which has been moved three-dimensionally by the view operation, the optimum photographed image or priority order is determined from among the multiple photographed images searched for based on the specified position, and some or all of the photographed images based on the determined optimum photographed image or priority order (hereinafter referred to as "optimum photographed image, etc.") can be displayed on the display unit 30, making it possible to easily check the photographed image of the desired inspection location.

[0112] FIG. 13 is a diagram showing an example of a screen 30A of the display unit 30 on which an optimal photographed image 100 determined from among a plurality of photographed images is displayed.

[0113] In this example, the CPU 20 functioning as the display switching unit 20C switches the function of the display control unit 26 from the first display control unit to the second display control unit when displaying the optimal captured image 100 on the display unit 30. Switching of the display from the three-dimensional model 16B to the optimal captured image 100 by the display switching unit 20C can be performed automatically when the optimal captured image 100 is determined by the captured image determination unit 20B and display of the optimal captured image 100 becomes possible.

[0114] FIG. 14 is a diagram showing another example of the screen 30A of the display unit 30 on which the optimal photographed image 100 determined from among a plurality of photographed images is displayed.

[0115] 14, one captured image 100 is displayed on the screen 30A of the display unit 30, and thumbnails 102 of the multiple captured images are displayed at the bottom of the screen 30A. The thumbnails 102 of the multiple captured images are arranged according to the priority order of the multiple captured images, and the optimal captured image 100 (the captured image with the highest priority order) among the multiple captured images is displayed first.

[0116] The user can display on the screen 30A of the display unit 30 the original image (captured image) corresponding to the desired thumbnail 102 by selecting the desired thumbnail 102 from the thumbnails 102 arranged according to the priority order.

[0117] FIG. 15 is a diagram showing yet another example of the screen 30A of the display unit 30 on which the optimal captured image etc. determined from among a plurality of captured images is displayed.

[0118] 15, a plurality of captured images 100 are displayed on a screen 30A of the display unit 30, and a scroll bar 36 is displayed at the bottom of the screen 30A. Priorities are assigned to the plurality of captured images, and the plurality of captured images including the captured image 100 with the highest priority are displayed first.

[0119] If the user wishes to see a captured image that is not displayed on the screen 30A of the display unit 30, the user can operate the scroll bar 36 with a mouse or the like to move (scroll) the captured image 100 displayed on the screen 30A of the display unit 30, thereby viewing the desired captured image.

[0120] Returning to FIG. 12, when the optimum photographed image, etc. is displayed on the display unit 30, the CPU 20 then determines whether or not to switch the display on the display unit 30 from the optimum photographed image, etc. to the three-dimensional model 16B (step S30). The determination of switching from the optimum photographed image, etc. to the three-dimensional model 16B can be made based on a user operation on the operation unit 18. For example, when the user wants to check photographed images of a different inspection location, the user can perform a switching operation on the operation unit 18 to switch from the display of the optimum photographed image, etc. to the display of the three-dimensional model 16B.

[0121] In step S30, when switching from the optimal photographed image or the like to the display of the three-dimensional model 16B (if "Yes"), the CPU 20 transitions to step S10.

[0122] This allows the 3D model 16B showing a full view of the bridge to be displayed on the display unit 30 (see FIG. 5). When displaying the 3D model 16B on the display unit 30, the CPU 20 functioning as the display switching unit 20C can switch the function of the display control unit 26 from the second display control unit to the first display control unit.

[0123] When switching from the optimum photographed image or the like to the display of the three-dimensional model 16B (when "Yes"), the CPU 20 may transition to step S12. This makes it possible to display the most recent three-dimensional model 16B when switching the display from the three-dimensional model 16B to the optimum photographed image or the like, which is preferable when checking a photographed image of an inspection location close to the previous inspection location.

[0124] On the other hand, if it is determined in step S30 that the display should not be switched from the optimal photographed image etc. to the three-dimensional model 16B (if "No"), the process proceeds to step S32.

[0125] In step S32, the CPU 20 determines whether or not an instruction to end the image display has been input from the operation unit 18, and if the instruction to end has not been input (if "No"), the process returns to step S28 and continues to display the management information on the display unit 30. On the other hand, if an instruction to end has been input (if "Yes"), the process relating to the image display is terminated.

[0126] In this example, the display of the three-dimensional model 16B and the display of the optimal photographed image, etc. on the display unit 30 are switched based on a user operation, etc. However, without being limited thereto, the first display control unit may display the three-dimensional model 16B in the first display area of ​​the display unit 30, and the second display control unit may display the optimal photographed image, etc. in the second display area of ​​the display unit 30, so that both are displayed simultaneously.

[0127] Incidentally, since the three-dimensional model 16B in this example is a polygon obtained by texture mapping a photographed image onto a polygonal shape, by enlarging and displaying the three-dimensional model 16B, it is possible to confirm the surface properties of the structure to some extent, but it is not possible to confirm small damage (for example, cracks with a width of 0.1 mm, etc.) This is because there is a limit to the amount of data in the three-dimensional model 16B, and even if the three-dimensional model 16B is enlarged, it is not possible to visually recognize it as the original photographed image or an image equivalent thereto.

[0128] In contrast, by using the 3D model 16B of the structure, the desired inspection location can be easily identified, and by displaying the original photographed image (optimal photographed image, etc.) corresponding to the identified inspection location, damage, etc. at the inspection location can be easily confirmed.

[0129] <Second embodiment> FIG. 16 is a flowchart showing a second embodiment of the image display method according to the present invention.

[0130] In FIG. 16, a second embodiment of the image display method according to the present invention will be described while explaining the operation of each part of the image display device 10 shown in FIG. 1 and FIG.

[0131] The image display method of the second embodiment shown in FIG. 16 is a method in which, when a desired captured image is selected from a group of captured images 16A, it is possible to easily confirm to which position on the three-dimensional model 16B of the structure the selected captured image corresponds.

[0132] In FIG. 16, display control unit 26 functioning as a third display control unit reads image list 16C from storage unit 16, and displays read image list 16C on display unit 30 (step S100, third display step).

[0133] 11, the image list 16C includes attributes of each captured image in the captured image group 16A, information for managing each captured image, information for linking each captured image to the three-dimensional model 16B, etc. This image list 16C can be created by the CPU 20 functioning as the image list creation section 20D based on the captured image group 16A and various information linked to the captured image group 16A, but it may also be created by an external device and stored in the storage section 16.

[0134] Next, the process of step S100 will be described in detail.

[0135] FIG. 17 is a diagram showing a subroutine illustrating a first example of the process of step S100.

[0136] In FIG. 17, the CPU 20 functioning as the condition setting section 30F sets conditions (narrowing conditions) indicative of narrowing down the captured images to be extracted from the captured image group 16A in response to a user operation on the operation section 18 (step S101).

[0137] Here, the narrowing down conditions include information indicating whether the captured image is linked to the inspection record of the structure, information indicating whether it is linked to the damage (crack) detection result, and information indicating whether it is linked to the repair record (repair drawing).

[0138] The narrowing conditions are not limited to the above examples, and may be, for example, information indicating the management classification of a structure. As described above, regular inspections of a structure are carried out for each management classification, which is a classification of inspection units for the parts and members that make up the structure, and in the case of a bridge, the main girders, cross beams, deck (coffering that makes up the deck), piers, etc. correspond to the management classifications. Information indicating these management classifications can be used as narrowing conditions.

[0139] The CPU 20 functioning as the image list creation section 20D narrows down the captured images in the captured image group 16A based on the narrowing down conditions set by the condition setting section 20F, and creates an image list showing the narrowed down captured images (step S102).

[0140] The display control unit 26 functioning as the third display control unit causes the display unit 30 to display an image list of the captured images narrowed down by the narrowing down conditions (step S103). As a result, the image list displayed on the display unit 30 is updated according to the narrowing down conditions set by the user.

[0141] For example, by setting information indicating whether the images are linked to the inspection records of the structures as the filtering conditions, an image list consisting of only the captured images linked to the inspection records can be displayed on the display unit 30. Furthermore, by setting information indicating the management classification of the structures as the filtering conditions, an image list consisting of only the captured images linked to the set management classification (captured images in which the area of ​​the structures corresponding to the set management classification is captured) can be displayed on the display unit 30.

[0142] FIG. 18 is a diagram showing a subroutine illustrating a second example of the process of step S100.

[0143] In FIG. 18, the CPU 20 functioning as the condition setting section 30F sets a condition (sorting condition) indicating rearrangement of the captured image group 16A in response to a user operation on the operation section 18 (step S104).

[0144] Here, the sorting conditions include the degree of facing of the photographed image to the subject (structure), the distance of the photographed image to the structure, and the photographed date and time (see FIG. 11).

[0145] The sorting conditions are not limited to the above example, and may be other sorting conditions such as the clarity of each captured image, but it is preferable that the image list 16C has at least information corresponding to the sorting conditions.

[0146] The CPU 20 functioning as the image list creation section 20D creates an image list by rearranging the captured image group 16A based on the sorting conditions set by the condition setting section 20F (step S105).

[0147] The display control section 26 functioning as the third display control section causes the display section 30 to display an image list of the captured image group 16A rearranged according to the sorting conditions (step S106, third display step).

[0148] For example, an image list 16C (original image list) stored in the memory unit 16 can be displayed on the display unit 30, and by clicking on a desired item (sorting condition) on the displayed image list, an image list of the captured image group 16A sorted in ascending or descending order according to the sorting condition can be displayed on the display unit 30.

[0149] Fig. 19 is a diagram showing a subroutine showing a third example of the process of step S100. In Fig. 19, the same step numbers are used for the parts common to the processes of the first and second examples shown in Fig. 17 and Fig. 18, and detailed descriptions thereof will be omitted.

[0150] The third example shown in FIG. 19 is a combination of the first and second examples shown in FIG. 17 and FIG. 18, and first, the captured images in the captured image group 16A are narrowed down based on the narrowing down conditions (steps S101 and S102).

[0151] Next, an image list is created by sorting the narrowed-down captured images based on the sorting conditions (steps S104 and S105), and the created image list (the image list of the narrowed-down and sorted captured images) is displayed on the display unit 30 (step S107).

[0152] 16, CPU 20 determines whether or not to end the display of the image list displayed on display unit 30 (step S108). CPU 20 determines whether or not an instruction to end the image display has been input from operation unit 18, and if an instruction to end the image display has not been input (if "No"), CPU 20 transitions to step S110, and if an instruction to end the image display has been input (if "Yes"), CPU 20 ends the process related to the image display.

[0153] In step S110, the CPU 20 determines whether or not an instruction to select an arbitrary captured image from the image list has been accepted in response to a user operation on the operation unit 18 functioning as a third operation unit.

[0154] If it is determined that a selection instruction has not been accepted (if "No"), the process returns to step S100, the image list continues to be displayed on the display unit 30, and an instruction to select a captured image can be accepted.

[0155] On the other hand, if it is determined that the selection instruction has been received (if "Yes"), the CPU 20 functioning as the position information acquisition unit 20E acquires the position information on the 3D model 16B corresponding to the selected photographed image (step S120). This position information can be read from the image list 16C based on the image file name of the selected photographed image.

[0156] Next, the display control unit 26 functioning as a fourth display control unit switches the display on the display unit 30 from the image list 16C to the three-dimensional model 16B, etc. That is, the display control unit 26 reads out the three-dimensional model 16B stored in the storage unit 16 and displays it on the display unit 30, and also superimposes an index indicating the position of the photographed image selected and instructed on the three-dimensional model 16B displayed on the display unit 30 based on the position information on the three-dimensional model 16B acquired by the position information acquisition unit (step S130, fourth display step).

[0157] FIG. 20 is a diagram showing an example of a screen 30A of the display unit 30 on which a selectively instructed photographed image 100 and a three-dimensional model 16B including the photographed range of the photographed image 100 are displayed.

[0158] 20, in order to make it easy to confirm the position of the photographed image 100 on the three-dimensional model 16B, the three-dimensional model 16B is automatically enlarged, translated, and rotated to display a part of the three-dimensional model 16B. Note that the view operation such as enlarging the three-dimensional model 16B can be automatically performed based on the position information of the structure shown in the photographed image 100 recorded in the image list 16C (for example, the three-dimensional positions of the four corners of the photographed image 100).

[0159] 20, the captured image 100 and the three-dimensional model 16B are displayed side by side on the screen 30A of the display unit 30, and an index 100A indicating the position of the captured image 100 is superimposed on the three-dimensional model 16B. This index 100A is displayed as a frame indicating the shooting range of the captured image 100. The frame indicating the shooting range of the captured image 100 can be formed by line segments connecting the three-dimensional positions of the four corners of the captured image 100.

[0160] Furthermore, if the user is unable to grasp the position of the captured image 100 within the entire three-dimensional model 16B from the display of the portion of the three-dimensional model 16B shown in Figure 20, the user can manually view (zoom out) the three-dimensional model 16B so that a wider range of the three-dimensional model 16B (see Figure 6) is displayed, or manually view the three-dimensional model 16B so that a full view of the three-dimensional model 16B (see Figure 5) is displayed.

[0161] When a wide-range three-dimensional model 16B is displayed, it is preferable that the indicator indicating the position of the photographed image 100 to be superimposed and displayed on the three-dimensional model 16B be a high-luminance point or a blinking high-luminance point.

[0162] Conversely to the above example, first, an index indicating the position of the photographed image 100 is superimposed on the three-dimensional model 16B, and the entire view of the three-dimensional model 16B is displayed on the display unit 30 (see FIG. 5). In this case, it is preferable to automatically rotate and move the three-dimensional model 16B so that the index can be confirmed. This allows the user to roughly grasp which part of the structure the photographed image 100 is taken of.

[0163] If the user wishes to grasp in detail which part of the structure the captured image 100 depicts, the user can manually perform a view operation to enlarge the three-dimensional model 16B (a view operation to translate and / or rotate as necessary) to cause the display unit 30 to display the enlarged three-dimensional model 16B, that is, a three-dimensional model 16C, with an indicator indicating the position of the captured image 100 superimposed thereon (see Figures 6 and 20).

[0164] The user can easily understand which position of the structure (3D model 16B) the image selected from the image list was captured by using the indicator indicating the position of the captured image superimposed on the 3D model 16C as described above.

[0165] 16, the CPU 20 functioning as the display switching unit 20C determines whether or not to switch the display on the display unit 30 from the display of the 3D model 16B, etc., to the display of the image list 16C (step S140). The determination to switch to the image list 16C can be made based on a user operation on the operation unit 18. For example, when the user wants to select a different captured image, the user can perform a switching operation on the operation unit 18 to switch to the display of the image list 16C.

[0166] In step S140, when switching from displaying the three-dimensional model 16B etc. to displaying the image list 16C (if "Yes"), the CPU 20 transitions to step S100.

[0167] This allows image list 16C to be displayed on display unit 30. When CPU 20 functioning as display switching unit 20C causes image list 16C to be displayed on display unit 30, it can switch the function of display control unit 26 from the fourth display control unit to the third display control unit.

[0168] On the other hand, if it is determined in step S140 that the display should not be switched to the image list 16C (if "No"), the CPU 20 transitions to step S150.

[0169] In step S150, the CPU 20 determines whether or not an instruction to end image display has been input from the operation unit 18, and if there has been no input to end the image display (if "No"), the process returns to step S130 and continues to display the three-dimensional model 16B, etc. on the display unit 30. On the other hand, if there has been an input to end the image display (if "Yes"), the process relating to the image display is terminated.

[0170] In this example, the display of the image list 16C and the three-dimensional model 16B, etc. on the display unit 30 are switched based on user operation, etc., but the display of the image list 16C and the three-dimensional model 16B, etc. may also be displayed simultaneously on the display unit 30.

[0171] [others] The user may switch between the image display method of the first embodiment and the image display method of the second embodiment as appropriate.

[0172] Furthermore, the three-dimensional model can be generated by using a group of captured images of a subject, and is not limited to being generated by the SfM technique, but can be generated by various other methods.

[0173] For example, a three-dimensional model of a structure can be generated by acquiring three-dimensional information of a subject from two parallax images captured by a twin-lens camera, or a three-dimensional model can be generated by acquiring a photographed image of a subject using a time-of-flight camera and acquiring three-dimensional coordinates of the subject corresponding to each pixel on the photographed image.Furthermore, a laser scanner having a camera function can be included, and a three-dimensional model of a structure can be generated based on the three-dimensional information of the structure acquired by the laser scanner.

[0174] The hardware for realizing the image display device according to the present invention can be composed of various processors. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes a program and functions as various processing units, a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacture such as an FPGA (Field Programmable Gate Array), and a dedicated electric circuit, which is a processor having a circuit configuration designed specifically to execute a specific process such as an ASIC (Application Specific Integrated Circuit). One processing unit constituting the image display device may be composed of one of the various processors described above, or may be composed of two or more processors of the same or different types. For example, one processing unit may be composed of multiple FPGAs, or a combination of a CPU and an FPGA. In addition, multiple processing units may be composed of one processor. As an example of configuring multiple processing units with one processor, first, there is a form in which one processor is configured by a combination of one or more CPUs and software, as represented by a computer such as a client or server, and this processor functions as multiple processing units. Secondly, there is a form using a processor that realizes the functions of the entire system including multiple processing units on a single IC (Integrated Circuit) chip, as typified by System On Chip (SoC). In this way, the various processing units are configured as a hardware structure using one or more of the above-mentioned various processors. Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

[0175] The present invention also includes an image display program that, when installed in a computer that can access a storage unit that stores a three-dimensional model of a subject and a group of captured images of the subject by changing the shooting position and shooting direction with respect to the subject, causes the computer to function as an image display device according to the present invention, and a storage medium on which this image display program is recorded.

[0176] Furthermore, the present invention is not limited to the above-described embodiment, and it goes without saying that various modifications are possible without departing from the spirit of the present invention. [Explanation of symbols]

[0177] 1 Bridges 2 Main digit 3 crossbeam 4. Anti-tilt structure 5 Horizontal structure 6 Floor slab 7. Pier 10 Image display device 12 Image acquisition section 14C Image List 15C Image List 16 Memory section 16A Image Group 16B 3D model 16C Image List 18 Control section 20 CPU 20A Image search section 20B Shooting image determination unit 20C Display switching section 20D Image list creation section 20E Location information acquisition section 20F Condition setting section 22 RAM 24 ROM 26 Display control section 30 Display section 30A screen 30F Condition setting section 32 Cursor 34 indicators 36 Scrollbar 100 images 100A indicator 102 Thumbnails S10~S150 steps

Claims

1. A storage unit that stores a three-dimensional model of a subject, which is a structure to be inspected, and a group of captured images of the subject, the captured images being obtained by changing the photographing position and photographing direction of the subject; a first operation unit that receives a view operation of the three-dimensional model to be displayed on the display unit when the three-dimensional model stored in the storage unit is to be displayed on the display unit; a first display control unit that reads out the three-dimensional model stored in the storage unit and displays it on the display unit, and displays the three-dimensional model based on a view operation accepted by the first operation unit; a second operation unit that receives, when a position on the screen of the display unit on the three-dimensional model displayed on the display unit is designated in response to a user operation, position information indicating the designated position; a captured image search unit that identifies a three-dimensional position on the three-dimensional model based on the received position information, and searches the captured images stored in the storage unit for a plurality of captured images including a pixel corresponding to the identified three-dimensional position; a photographed image determination unit that determines an optimal photographed image from the retrieved photographed images or determines a priority order of the retrieved photographed images; a second display control unit that causes the display unit to display the optimum photographed image determined by the photographed image determination unit, or causes the display unit to display a part or all of the plurality of photographed images in accordance with the determined priority order; An image display device comprising:

2. 2. The image display device according to claim 1, wherein the captured image determination unit determines the optimal captured image or determines a priority order of the retrieved captured images based on one or more conditions of the degree of orientation of the captured image relative to the subject and the amount of deviation between the position on the captured image corresponding to the identified three-dimensional position and the center position of the captured image.

3. the view operation is an operation of enlarging, reducing, translating, or rotating the three-dimensional model to be displayed on the display unit, The image display device according to claim 1 , wherein the first display control unit enlarges, reduces, translates, or rotates the three-dimensional model to be displayed on the display unit based on the view operation.

4. 4. The image display device according to claim 1, wherein the three-dimensional model is represented by a three-dimensional point cloud consisting of three-dimensional information of a large number of points on the surface of the subject, a representation of the surface of the subject as a collection of polygons based on the three-dimensional point cloud, or a texture mapping of an image of the subject onto the polygons.

5. the first display control unit causes the three-dimensional model to be displayed in a first display area of ​​the display unit; The image display device according to claim 1 , wherein the second display control unit causes the captured image to be displayed in a second display area of ​​the display unit.

6. 5. The image display device according to claim 1, further comprising a display switching unit that switches between displaying the three-dimensional model by the first display control unit on the display unit and displaying the captured image by the second display control unit.

7. a third display control unit that causes an image list indicating the group of captured images to be displayed on a display unit; a third operation unit that accepts an instruction to select an arbitrary captured image from the image list by a user operation; a position information acquisition unit that acquires position information on the three-dimensional model corresponding to the selected photographed image; a fourth display control unit that reads out the three-dimensional model stored in the storage unit and displays it on the display unit, and that superimposes and displays an index indicating the position of the selected captured image on the three-dimensional model displayed on the display unit based on position information on the three-dimensional model acquired by the position information acquisition unit; and The image display device according to claim 1 , further comprising:

8. a condition setting unit that sets a condition for narrowing down the captured images to be extracted from the group of captured images and / or for rearranging the group of captured images; an image list creation unit that creates the image list indicating captured images narrowed down from the group of captured images based on the conditions set by the condition setting unit, and / or the image list in which the group of captured images is rearranged; The image display device according to claim 7 , wherein the third display control unit updates the image list displayed on the display unit with the image list created by the image list creation unit.

9. The image display device according to claim 7 or 8, wherein the items displayed in the image list include one or more of a reduced image of each captured image in the group of captured images, identification information identifying each captured image, the orientation of the captured image relative to the subject, the distance of the captured image relative to the subject, the clarity of the captured image, information indicating whether the captured image is linked to an inspection record of the subject, a damage detection result or a repair record, and the date and time of capture.

10. 10. The image display device according to claim 7, wherein the fourth display control unit causes the three-dimensional model with the indicator superimposed thereon to be displayed in a first display area of ​​the display unit, and causes the selected captured image to be read out from the memory unit and displayed in a second display area of ​​the display unit.

11. 11. The image display device according to claim 7, wherein the fourth display control unit enlarges, translates, or rotates the three-dimensional model to be displayed on the display unit based on the acquired position information on the three-dimensional model, thereby making the indicator superimposed on the three-dimensional model easier to see.

12. A step of preparing a storage unit that stores a three-dimensional model of a subject, which is a structure to be inspected, and a group of photographed images of the subject, the group being photographed by changing a photographing position and a photographing direction of the subject; a step of receiving, by a first operation unit, a view operation of the three-dimensional model to be displayed on a display unit when the three-dimensional model stored in the storage unit is to be displayed on the display unit; a first display step in which a first display control unit reads out the three-dimensional model stored in the storage unit and displays it on the display unit, and displays the three-dimensional model based on a view operation accepted by the first operation unit; a step of receiving, by a second operation unit, position information indicating a specified position on a screen of the display unit when a position on the three-dimensional model displayed on the display unit is specified in response to a user operation; a captured image search unit specifying a three-dimensional position on the three-dimensional model based on the received position information, and searching for a plurality of captured images including a pixel corresponding to the specified three-dimensional position from the captured image group stored in the storage unit; a photographed image determination unit determining an optimal photographed image from the retrieved photographed images or determining a priority order of the retrieved photographed images; a second display step in which a second display control unit causes the display unit to display the determined optimum captured image, or causes the display unit to display a part or all of the plurality of captured images in accordance with the determined priority order; An image display method comprising:

13. The image display method according to claim 12, wherein the step of determining the captured image determines the optimal captured image or determines the priority order of the retrieved captured images based on one or more conditions of the degree of orientation of the captured image relative to the subject and the amount of deviation between the position on the captured image corresponding to the identified three-dimensional position and the center position of the captured image.

14. the view operation is an operation of enlarging, reducing, translating, or rotating the three-dimensional model to be displayed on the display unit, 14. The image display method according to claim 12, wherein the first display step enlarges, reduces, translates, or rotates the three-dimensional model displayed on the display unit based on the view operation.

15. 15. The image display method according to claim 12, wherein the three-dimensional model is represented by a three-dimensional point cloud consisting of three-dimensional information of a large number of points on the surface of the subject, a representation of the surface of the subject as a collection of polygons based on the three-dimensional point cloud, or texture mapping of an image of the subject onto the polygons.

16. The first display step displays the three-dimensional model in a first display area of ​​the display unit, The image display method according to claim 12 , wherein the second display step displays the captured image in a second display area of ​​the display section.

17. The image display method according to claim 12 , wherein a display switching unit switches between displaying the three-dimensional model in the first display step and displaying the captured image in the second display step on the display unit.

18. a third display step in which a third display control unit causes the display unit to display an image list indicating the group of captured images; a step of receiving an instruction by a third operation unit to select an arbitrary captured image from the image list by a user operation; a position information acquiring unit acquiring position information on the three-dimensional model corresponding to the selected photographed image; a fourth display step in which a fourth display control unit reads out the three-dimensional model stored in the storage unit and displays it on the display unit, and a fourth display step in which an index indicating the position of the selected photographed image is superimposed on the three-dimensional model displayed on the display unit based on the acquired position information on the three-dimensional model; The image display method according to any one of claims 12 to 17, comprising:

19. setting a condition indicating narrowing down of captured images to be extracted from the group of captured images and / or rearrangement of the group of captured images by a condition setting unit; an image list creation unit creating the image list indicating captured images narrowed down from the group of captured images based on the set condition, and / or creating the image list by rearranging the group of captured images; 20. The image display method according to claim 18, wherein the third display step updates the image list displayed on the display unit with the created image list.

20. The image display method according to claim 18 or 19, wherein the items displayed in the image list include one or more of a reduced image of each captured image in the group of captured images, identification information identifying each captured image, the orientation of the captured image relative to the subject, the distance of the captured image relative to the subject, the clarity of the captured image, information indicating whether the captured image is linked to an inspection record, damage detection result or repair record of the subject, and the date and time of capture.

21. 21. The image display method according to claim 18, wherein the fourth display step displays the three-dimensional model with the indicator superimposed thereon in a first display area of ​​the display unit, and reads out the selected captured image from the storage unit and displays it in a second display area of ​​the display unit.

22. 22. The image display method according to claim 18, wherein the fourth display step enlarges, translates, or rotates the three-dimensional model to be displayed on the display unit based on the acquired position information on the three-dimensional model, thereby making the indicator superimposed on the three-dimensional model easier to see.

23. An image display program installed in a computer that can access a storage unit that stores a three-dimensional model of a subject that is a structure to be inspected and a group of captured images of the subject by changing a photographing position and a photographing direction of the subject, a function of accepting a view operation of the three-dimensional model when the three-dimensional model stored in the storage unit is displayed on a display unit; a function of reading out the three-dimensional model stored in the storage unit, displaying the three-dimensional model on the display unit, and displaying the three-dimensional model based on the view operation; a function of receiving, when a position on the screen of the display unit on the three-dimensional model displayed on the display unit is designated in response to a user operation, position information indicating the designated position; a function of identifying a three-dimensional position on the three-dimensional model based on the received position information, and searching for a plurality of captured images including a pixel corresponding to the identified three-dimensional position from the group of captured images stored in the storage unit; a function of determining an optimal photographed image from the retrieved plurality of photographed images, or determining a priority order of the retrieved plurality of photographed images; a function of displaying the determined optimal photographed image on the display unit, or displaying a part or all of the plurality of photographed images on the display unit in accordance with the determined priority order; An image display program that causes the computer to realize the above.

24. a function of displaying an image list indicating the group of captured images on the display unit; a function of receiving an instruction to select a captured image selected from the image list by a user operation; a function of acquiring position information on the three-dimensional model corresponding to the selected photographed image; a function of reading out the three-dimensional model stored in the storage unit and displaying it on the display unit, the function of superimposing an index indicating the position of the selected captured image on the three-dimensional model displayed on the display unit based on the acquired position information on the three-dimensional model; 24. The image display program according to claim 23, which causes the computer to execute the steps of:

24. The image display program according to claim 23,

Images