Image processing apparatus, image processing method, and program

Abstract

To enable a user to easily determine a change in a phenomenon, such as a crack or damage that has occurred in a structure.SOLUTION: An image processing apparatus includes: a posture estimation section that estimates a position and a posture of a camera when the camera has captured each of at least two images including the same phenomena on the basis of at least two images and three-dimensional data of a structure; an area selection section that selects an area of the phenomenon in each of the at least two images; a three-dimensional position identification section that identifies a position of each of the phenomena in a coordinate system of the three-dimensional data on the basis of the estimated position and posture of the camera and the area selected in each of the at least two images; and a projection section that generates a projection image by projecting the three-dimensional data including the identified position of each of the phenomena and compositing the phenomena in such a manner that the phenomena for the images are distinguishable from each other.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] As a related technique, Patent Document 1 discloses an image processing device used for inspecting structures such as bridges. In Patent Document 1, an inspector photographs cracked parts of a structure from multiple observation points in order to observe or inspect the state of cracks in the structure. The inspector inputs the multiple captured two-dimensional images into an image processing device.

[0003] The image processing device generates a three-dimensional mesh that approximates at least a portion of the structure. The image processing device also generates a two-dimensional development by developing the three-dimensional mesh onto a two-dimensional plane. The two-dimensional development is a diagram transformed so that each partial plane included in the three-dimensional mesh is orthogonally projected. The image processing device performs coordinate transformation of the captured multiple images with respect to the two-dimensional development, and generates a panoramic image by synthesizing the coordinate-transformed multiple images based on the coordinate values.

[0004] In the patent, a user can refer to one of a number of two-dimensional images and input coordinate values ​​of an object such as a crack and text indicating an annotation for the object. In the patent, the stored annotation can be displayed by being superimposed on another image having panoramic coordinates. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2017-168077 A Summary of the Invention [Problem to be solved by the invention]

[0006] In Patent Document 1, an image processing device synthesizes a plurality of images taken at the same time into a panorama. In this case, a user such as an inspector can check cracks that have occurred in a bridge using the plurality of images. However, Patent Document 1 does not take into consideration checking whether damage such as cracks has changed over a period of several months or years. In Patent Document 1, since a plurality of images are simply synthesized into a panorama, a user cannot easily determine changes in cracks or scratches that have occurred between images.

[0007] In view of the above circumstances, an object of the present disclosure is to provide an image processing device, an image processing method, and a program that enable a user to easily determine changes in phenomena such as cracks or damage that have occurred in a structure. [Means for solving the problem]

[0008] An image processing device according to a first aspect of the present disclosure includes: an attitude estimation unit that estimates a position and attitude of a camera at the time of capturing each of two or more images, each of which includes the same event, based on three-dimensional data of a structure; an area selection unit that selects an area of ​​the event in each of the two or more images; a three-dimensional position identification unit that identifies a position of the event in a coordinate system of the three-dimensional data based on the estimated position and attitude of the camera and the area selected in each of the two or more images; and a projection unit that projects three-dimensional data including the position of the identified event, and generates a projected image in which the event is combined in a manner that allows it to be distinguished for each image.

[0009] An image processing method according to a second aspect of the present disclosure includes estimating a camera position and orientation at the time of capturing each of two or more images, each of which includes the same event, based on three-dimensional data of a structure, selecting an area of ​​the event in each of the two or more images, identifying a position of the event in a coordinate system of the three-dimensional data based on the estimated camera position and orientation and the area selected in each of the two or more images, projecting three-dimensional data including the identified position of the event, and generating a projected image in which the event is combined in a manner that allows it to be distinguished for each of the images.

[0010] A program according to a third aspect of the present disclosure is intended to cause a computer to execute a process including: estimating a camera position and orientation at the time when each of two or more images was taken based on two or more images each containing the same event and three-dimensional data of a structure; selecting an area of ​​the event in each of the two or more images; identifying a position of the event in a coordinate system of the three-dimensional data based on the estimated camera position and orientation and the area selected in each of the two or more images; projecting three-dimensional data including the identified position of the event; and generating a projected image in which the event is synthesized in a manner that enables it to be distinguished for each of the images. Effect of the Invention

[0011] The image processing device, image processing method, and program according to the present disclosure can enable a user to easily determine changes in phenomena such as cracks or damage that have occurred in a structure. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a block diagram showing an example of a schematic configuration of an image processing device according to the present disclosure. [Diagram 2] 1 is a block diagram showing an example of a schematic configuration of a first image processing device according to the present disclosure. [Diagram 3] FIG. 4 is a diagram showing an example of an image acquired by an image input unit; [Figure 4]FIG. 13 is a diagram showing an example of another image acquired by the image input unit. [Diagram 5] FIG. 2 is a diagram showing a schematic diagram of an event whose position is identified in three-dimensional data. [Figure 6] FIG. 10 is a diagram illustrating an event position identified on three-dimensional data and a projection range. [Figure 7] FIG. 13 is a diagram showing an example of a projected image to be displayed. [Figure 8] 10 is a flowchart showing an example of an operation procedure in the image processing device. [Figure 9] FIG. 2 is a block diagram illustrating an example of a schematic configuration of a second image processing device according to the present disclosure. [Figure 10] FIG. 13 is a diagram illustrating an example of information stored in an event DB. [Figure 11] FIG. 1 is a block diagram showing an example of the configuration of a computer device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Prior to describing the embodiments of the present disclosure, an overview of the present disclosure will be described. Fig. 1 shows an example of a schematic configuration of an image processing device according to the present disclosure. The image processing device 10 includes a posture estimation unit 11, a region selection unit 12, a three-dimensional position identification unit 13, and a projection unit 14.

[0014] The attitude estimation unit 11 estimates the position and attitude of the camera at the time of capturing each of the two or more images based on two or more images each including the same event and the three-dimensional data of the structure. The area selection unit 12 selects an area of ​​the event in each of the two or more images. The three-dimensional position identification unit 13 identifies the position of the event in the coordinate system of the three-dimensional data based on the estimated position and attitude of the camera and the area selected in each of the two or more images. The projection unit 14 projects the three-dimensional data including the identified event position, and generates a projected image in which the events are combined in a manner that allows them to be distinguished from one another in each image.

[0015] In the image processing device according to the present disclosure, the posture estimation unit 11 estimates the position and posture of the camera when the image is captured. The three-dimensional position identification unit 13 identifies the position of the event in the coordinate system of the three-dimensional data based on the estimated position and posture of the camera and the area of ​​the event selected in each image by the area selection unit 12. The projection unit 14 generates a projection image in which the position of the event on the three-dimensional data is projected, and the events of each image are synthesized in a manner that makes it possible to distinguish between the images. In the present disclosure, events such as cracks contained in each of the multiple images are projected onto the projection image. Therefore, a user can easily determine changes in events such as cracks and damage that have occurred in a structure by observing the projection image.

[0016] Hereinafter, an embodiment of the present invention will be described in detail. Note that the following description and drawings are omitted and simplified as appropriate for clarity of explanation. In addition, in each drawing, the same elements and similar elements are given the same reference numerals, and duplicate explanations are omitted as necessary.

[0017] A first embodiment will be described. FIG. 2 shows an example of a schematic configuration of a first image processing device according to the present disclosure. The image processing device 100 shown in FIG. 2 includes an image input unit 101, a camera attitude estimation unit 102, an event area selection unit 103, an event 3D (three dimensions) position identification unit 104, an event projection unit 105, and a 3D data DB (database) 120. In this embodiment, the image processing device 100 can be used for inspecting structures such as bridges. The inspection may be performed periodically, for example, every few months or every few years. Alternatively, the inspection may be performed irregularly.

[0018] The image processing device 100 may be physically configured as a device having, for example, one or more processors and one or more memories. At least a part of the functions of each unit in the image processing device 100 may be realized by one or more processors executing processes according to instructions read from one or more memories.

[0019] The image input unit 101 acquires a plurality of images. In this embodiment, the image input unit 101 acquires images of a structure such as a bridge. The image input unit 101 acquires, for example, one or more images captured at a first time period and one or more images captured at a second time period. For example, an inspector of a structure captures an event or abnormality such as a crack or damage occurring in the structure, and inputs the captured image to the image input unit 101. The images acquired by the image input unit 101 include two or more images captured at different times or periods, each of which includes the same event.

[0020] The 3D data DB 120 stores three-dimensional data of structures. The three-dimensional data is also called three-dimensional point cloud data. The three-dimensional data may be created using a distance measurement sensor such as LiDAR (Light Detection And Ranging). The camera posture estimation unit 102 estimates the position and posture of the camera at the time of shooting for each of two or more images that were shot at different times and include the same event, based on each image and the three-dimensional data of the structure. In other words, the camera posture estimation unit 102 estimates from what position and in what posture each image was shot. The camera posture estimation unit 102 corresponds to the posture estimation unit 11 shown in FIG. 1.

[0021] The event area selection unit 103 selects an event area, i.e., an area of ​​an event, in each of the multiple images acquired by the image input unit 101. The event area selection unit 103 may, for example, perform image processing on each image, automatically recognize areas where cracks or damage have occurred in a structure, and select the areas where cracks or damage have occurred as event areas. The event area selection unit 103 may, for example, use an AI (artificial intelligence) engine to recognize the areas where cracks or damage have occurred. The event area selection unit 103 corresponds to the area selection unit 12 shown in FIG. 1.

[0022] FIG. 3 shows an example of an image acquired by the image input unit 101. The image shown in FIG. 3 is assumed to be an image obtained by photographing a structure at a first time, for example. This image includes a deformation 150 such as a crack. The event area selection unit 103 selects the area of ​​the deformation 150 from the image, for example, using an AI engine. In addition, the camera attitude estimation unit 102 estimates the position and attitude of the camera when the image shown in FIG. 3 was photographed. The camera attitude estimation unit 102, for example, compares three-dimensional data of the structure stored in the 3D data DB 120 with each area of ​​the structure in the image, thereby estimating from what position and in what attitude the image was photographed.

[0023] FIG. 4 shows an example of another image acquired by the image input unit 101. The image shown in FIG. 4 is assumed to be an image obtained by photographing a structure at a second time, for example, one year after the first time. This image includes the same crack as the deformation 150 included in the image shown in FIG. 3. However, the image shown in FIG. 4 is photographed at a position and posture different from the position and posture of the camera when the image shown in FIG. 3 is photographed. The event area selection unit 103 selects the area of ​​the deformation 150 from the image, for example, using an AI engine. In addition, the camera posture estimation unit 102 estimates the position and posture of the camera when the image shown in FIG. 4 was photographed. A user such as an inspector may manually recognize the location where the crack or damage occurs by observing each of the images shown in FIG. 3 and FIG. 4, and manually set the area including the event in the event area selection unit 103.

[0024] Returning to FIG. 2, the event 3D position identification unit 104 identifies the position of the event in the coordinate system of the three-dimensional data, that is, the event position, based on the event region selected by the event region selection unit 103 and the position and orientation of the camera estimated by the camera orientation estimation unit 102. Hereinafter, for convenience, one of the multiple images including the same event is called the first image, and the other is called the second image. For example, the event 3D position identification unit 104 identifies the event position in the coordinate system of the three-dimensional data by mapping the event region selected in the first image to the three-dimensional data based on the position and orientation of the camera estimated for the first image. In addition, the event 3D position identification unit 104 identifies the event position in the coordinate system of the three-dimensional data by mapping the event region selected in the second image to the three-dimensional data based on the position and orientation of the camera estimated for the second image. The event 3D position identification unit 104 corresponds to the three-dimensional position identification unit 13 shown in FIG. 1.

[0025] The event projection unit 105 projects the three-dimensional data including the specified event position to generate a projection image. The event projection unit 105 may project the three-dimensional data by parallel projection, although this is not particularly limited. For example, the event projection unit 105 overlaps the events included in the multiple images so that the event included in the older image is at the top or bottom in the projection image.

[0026] In this embodiment, the event projection unit 105 synthesizes events included in each of the multiple images in a projected image in a manner that allows them to be distinguished from one another. For example, the event projection unit 105 assigns different display colors to each of the events included in each of the multiple images. Alternatively, the event projection unit 105 may overlay each of the events included in each of the multiple images with a slight shift. The event projection unit 105 may set a different transmittance for each of the events included in each of the multiple images. The event projection unit 105 corresponds to the projection unit 14 shown in FIG. 1.

[0027] FIG. 5 shows a schematic diagram of an event whose position has been specified in three-dimensional data. In FIG. 5, each point on the surface of the structure represents a data point of the three-dimensional data of the structure stored in the 3D data DB 120. The event 3D position specifying unit 104 specifies the position in the coordinate system of the three-dimensional data of the event region selected in the two-dimensional camera image based on the three-dimensional data of the structure and the estimated camera position and orientation. The event position in the coordinate system of the three-dimensional data specified by the event 3D position specifying unit 104 corresponds to the three-dimensional data of the event.

[0028] The event projection unit 105 determines the surface to which the event belongs, i.e., the surface on which the event occurs, as the projection surface from the position of the event in the coordinate system of the identified three-dimensional data. For example, in the three-dimensional data, the event projection unit 105 calculates the curvature of the surface around the event from the normal of each point of the three-dimensional data of the peripheral area of ​​the event 160, for example, a predetermined range including the event 160. In the three-dimensional data, the event projection unit 105 determines the surface having the same curvature as the calculated curvature in the peripheral area of ​​the predetermined range, i.e., in the area continuous with the predetermined range. For example, when an event occurs on a cylindrical bridge pier, the event projection unit 105 determines the surface of the cylinder as the projection surface.

[0029] FIG. 6 shows a schematic diagram of an event position and a projection range specified on three-dimensional data. The event projection unit 105 determines a projection range to be displayed in a projection image on the projection surface determined above. The event projection unit 105 determines, for example, a range represented by a rectangle of a predetermined size including the center position of the event in the coordinate system of the three-dimensional data, i.e., the center position of the event in the three-dimensional coordinate system, as the projection range. For example, the event projection unit 105 determines, as the projection range, a range represented by a rectangle of width w and height h with the center position of the event 160 as the projection center. The width w and height h may change depending on the size of the event. The event projection unit 105 generates, for example, an image of the event 160 viewed from the front of the projection surface as the projection image.

[0030] FIG. 7 shows an example of a projected image to be displayed. The event projection unit 105 displays the generated projected image on the display screen. The event projection unit 105 superimposes the event area selected in the first image and the event area selected in the second image in the projected image while shifting their positions. At this time, the event projection unit 105 superimposes the event selected in the first image and the event selected in the second image on the projected image in order based on the time or time when they were photographed. For example, the event projection unit 105 superimposes the event selected in the first image and the event selected in the second image on the projected image so that the event selected in the image photographed earlier among the first image and the second image is on top. The event projection unit 105 assigns different display colors to the event selected in the first image and the event selected in the second image in the projected image. In this case, the user can determine whether the event has changed during the period between the time when the first image was photographed and the time when the second image was photographed, for example, whether a crack has grown, by observing the projected image. The event projection unit 105 displays a scale indicating the scale in the projected image. By looking at the scale, the user can know the actual size of the crack or the like.

[0031] An operation procedure will be described. FIG. 8 shows an example of the operation procedure in the image processing device 100. The operation procedure of the image processing device 100 corresponds to an image processing method. The image input unit 101 acquires a plurality of images (step S1). In step S1, the image input unit 101 acquires, for example, a first image taken at a first time and a second image taken at a second time. The first image and the second image contain the same phenomenon, such as a crack. The image input unit 101 may acquire three or more images taken at three or more time periods containing the same phenomenon.

[0032] The event area selection unit 103 selects an event area in each of the multiple images acquired by the image input unit 101 (step S2). The camera attitude estimation unit 102 estimates the position and attitude of the camera at the time of capturing each image based on each image and the three-dimensional data of the structure stored in the 3D data DB 120 (step S3). In step S3, the camera attitude estimation unit 102 estimates the position and attitude of the camera at the time of capturing in the coordinate system of the three-dimensional data. Either step S2 or step S3 may be performed first. Alternatively, step S2 and step S3 may be performed in parallel.

[0033] The event 3D position identifying unit 104 identifies the three-dimensional position of each event, i.e., the event position in the coordinate system of the three-dimensional data, based on the event region selected in step S2 and the position and orientation of the camera estimated in step S3 (step S4). In step S4, the event 3D position identifying unit 104 identifies the position in the coordinate system of the three-dimensional data of the event region selected in the first image. Also, in step S4, the event 3D position identifying unit 104 identifies the position in the coordinate system of the three-dimensional data of the event region selected in the second image.

[0034] The event projection unit 105 projects the three-dimensional data including the event position identified in step S4 to generate a projection image (step S5). In step S5, the event projection unit 105 synthesizes the events included in each of the multiple images in the projection image in a manner that makes them distinguishable. For example, the event projection unit 105 assigns different display colors to the events included in each of the multiple images. The event projection unit 105 displays the generated projection image on a display screen.

[0035] In this embodiment, the camera posture estimation unit 102 estimates the position and posture of the camera at the time of capturing each image based on multiple images including the same event and the three-dimensional data of the structure. The event area selection unit 103 selects an event area in each of the multiple images. The event 3D position identification unit 104 identifies the three-dimensional position of each event based on the event area selected in each image and the position and posture of the camera estimated for each image. In this embodiment, the image processing device 100 estimates the position and posture of the camera at the time of capturing an image, and identifies the position of the event in the coordinate system of the three-dimensional data using the estimation result. In this way, the image processing device 100 can map events included in multiple images with different camera positions and postures at the time of capturing to the same three-dimensional position.

[0036] In this embodiment, the event projection unit 105 projects three-dimensional data including an event whose position in the coordinate system of the three-dimensional data is specified by the event 3D position specification unit 104, and generates a projection image. In other words, the event projection unit 105 generates a projection image that visualizes each event on the three-dimensional data. The event projection unit 105 synthesizes events included in each of a plurality of images in a manner that makes them distinguishable in the projection image. In this way, the image processing device in this embodiment can display events included in a plurality of images taken with different camera attitudes in a single projection image so that they can be compared. Therefore, a user can easily determine changes in events, such as cracks and damage that have occurred in a structure, for example changes over time, by observing the projection image.

[0037] Next, a second embodiment will be described. Fig. 9 shows an example of a schematic configuration of a second image processing device according to the present disclosure. The image processing device 100a shown in Fig. 9 has an event DB 130 in addition to the configuration of the image processing device 100 shown in Fig. 2. The event DB 130 stores, for each event and shooting time, information for identifying an image including the event, information on the event area in the image, and information on the estimated camera position and orientation.

[0038] FIG. 10 shows an example of information stored in the event DB 130. In FIG. 10, the event number represents information for identifying an event. In the example of FIG. 10, the event DB 130 stores information indicating which image each event was included in at each of the shooting times A, B, and C. For example, the event DB 130 stores information indicating that the event with the event number "1" is included in the image A-1 at the shooting time A, the image B-1 at the shooting time B, and the image C-1 at the shooting time C. The event DB 130 stores, for each image, information on the event area selected by the event area selection unit 103, and information on the position and attitude of the camera estimated by the camera attitude estimation unit 102. The event DB 130 may store, for each image, the position of the event in the coordinate system of the three-dimensional data identified by the event 3D position identification unit 104.

[0039] For example, it is assumed that the image input unit 101 acquires images A-1, B-1, and C-1, and image D-1 captured at capture time D. The camera posture estimation unit 102 refers to the event DB 130 and checks whether information on the position and posture of the camera estimated for each image is stored in the event DB 130. It is assumed that the event DB 130 stores information on the position and posture of the camera estimated for images A-1, B-1, and C-1. In this case, the camera posture estimation unit 102 acquires information on the position and posture of the camera estimated for those images from the event DB 130. If information on image D-1 is not stored in the event DB 130, the camera posture estimation unit 102 estimates the position and posture of the camera for image D-1.

[0040] Furthermore, the event area selection unit 103 refers to the event DB 130 and checks whether information on the event area selected for each image is stored in the event DB 130. It is assumed that the event DB 130 stores information on the event areas selected in the images A-1, B-1, and C-1. In this case, the event area selection unit 103 obtains information on the event areas in those images from the event DB 130. If information on the image D-1 is not stored in the event DB 130, the event area selection unit 103 selects an event area in the image D-1.

[0041] In this embodiment, event DB 130 stores information on the event area selected by event area selection unit 103, and information on the position and orientation of the camera estimated by camera orientation estimation unit 102. When information is stored in event DB 130, camera orientation estimation unit 102 and event area selection unit 103 acquire information from event DB 130. In this case, camera orientation estimation unit 102 and event area selection unit 103 can omit estimating the position and orientation of the camera and selecting an event area for images that have already been processed.

[0042] In this embodiment, the camera posture estimation unit 102 may estimate the position and posture of the camera for an unprocessed image. Also, the event area selection unit 103 may select an event area for an unprocessed image. In this manner, the processing load in the image processing device 100 may be reduced.

[0043] In the present disclosure, the image processing device 100 may be configured as a computer device or a server device. Fig. 11 shows an example of the configuration of a computer device that may be used as the image processing device 100. The computer device 500 has a processor (e.g., a central processing unit (CPU)) 510, a storage unit 520, a read only memory (ROM) 530, a random access memory (RAM) 540, a communication interface (IF) 550, and a user interface 560.

[0044] The communication interface 550 is an interface for connecting the computer device 500 to a communication network via a wired communication means or a wireless communication means, etc. The user interface 560 includes a display unit such as a display, and an input unit such as a keyboard, a mouse, and a touch panel.

[0045] The storage unit 520 is an auxiliary storage device that can hold various data. The storage unit 520 does not necessarily need to be a part of the computer device 500, and may be an external storage device or a cloud storage connected to the computer device 500 via a network.

[0046] The ROM 530 is a non-volatile storage device. For example, a semiconductor storage device having a relatively small capacity, such as a flash memory, is used for the ROM 530. The programs executed by the CPU 510 may be stored in the storage unit 520 or the ROM 530. The storage unit 520 or the ROM 530 stores various programs for implementing the functions of each unit of the image processing device 100.

[0047] The program includes a set of instructions (or software code) that, when loaded into a computer, causes the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, the computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, Compact Disc (CD), digital versatile disc (DVD), Blu-ray® disk or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, the transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[0048] The RAM 540 is a volatile storage device. Various semiconductor memory devices such as a dynamic random access memory (DRAM) or a static random access memory (SRAM) are used for the RAM 540. The RAM 540 can be used as an internal buffer for temporarily storing data and the like. The CPU 510 loads a program stored in the storage unit 520 or the ROM 530 into the RAM 540 and executes it. The CPU 510 executes the program to realize the functions of each unit in the image processing device 100. The CPU 510 may have an internal buffer for temporarily storing data and the like.

[0049] In the present disclosure, the image processing device 100 does not necessarily have to be configured as one device. The image processing device 100 may be configured using multiple devices that are physically separated. For example, the image processing device 100 may be configured using a first device that is portable and a second device that is installed in an office or the like. The first device has at least an image input unit 101 (see FIG. 2). In that case, a user may carry the first device to a site where a structure such as a bridge is located, and input a captured image at the site to the first device.

[0050] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above-mentioned embodiments. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Each embodiment can be appropriately combined with other embodiments.

[0051] Each drawing is merely an example for describing one or more embodiments. Each drawing may relate not only to one particular embodiment, but to one or more other embodiments. As can be understood by those skilled in the art, various features or steps described with reference to any one drawing may be combined with features or steps shown in one or more other drawings to create an embodiment not explicitly shown or described, for example. Not all features or steps shown in any one drawing are necessary to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0052] A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes.

[0053] [Appendix 1] a posture estimation unit that estimates a position and a posture of a camera at the time of capturing each of two or more images, each of which includes the same event, based on three-dimensional data of a structure; a region selection unit for selecting a region of the event in each of two or more images; a three-dimensional position identifying unit that identifies a position of the event in a coordinate system of the three-dimensional data based on the estimated camera position and orientation and a region selected in each of the two or more images; a projection unit that projects three-dimensional data including the positions of the identified events and generates projected images in which the events are synthesized in a manner that allows them to be distinguished from one another for each image.

[0054] [Appendix 2] The image processing device according to claim 1, wherein the projection unit changes at least one of a transmittance to be set and a display color to be assigned in the projected image for each event.

[0055] [Appendix 3] The image processing device according to claim 1 or 2, wherein the two or more images are taken at different times.

[0056] [Appendix 4] The image processing device according to claim 3, wherein the projection unit shifts the position of the event so that an image captured earlier is overlapped above or below in the projected image.

[0057] [Appendix 5] The image processing device according to any one of claims 1 to 4, wherein the projection unit determines a plane to which each event belongs as a projection plane based on the position of the event in the coordinate system of the identified three-dimensional data, and determines a projection range to be represented in the projection image on the determined projection plane.

[0058] [Appendix 6] The image processing device described in Appendix 5, wherein the projection unit calculates a curvature of a surface including a predetermined range including the position of the identified event from normals of each data point of the three-dimensional data of the predetermined range, and determines, as the projection surface, a surface in the three-dimensional data that has the same curvature as the calculated curvature in an area continuous with the predetermined range.

[0059] [Appendix 7] 7. The image processing device according to claim 5, wherein the projection unit determines, as the projection range, a range represented by a rectangle having a center position of the position of the identified event as a center.

[0060] [Appendix 8] The image processing device according to any one of claims 1 to 7, wherein the projection unit displays a scale in the projected image.

[0061] [Appendix 9] an event database for storing, for each of the events, information identifying an image in which the event is included, information on the area of ​​the event within the image, and information on the estimated position and orientation of the camera; the posture estimation unit acquires the information on the position and posture of the camera from the event database when the information on the position and posture of the camera is stored in the event database; The image processing device according to any one of claims 1 to 8, wherein the area selection unit acquires information on the area of ​​the event from the event database when information on the area of ​​the event is stored in the event database.

[0062] [Appendix 10] estimating a position and an orientation of a camera at the time of capturing each of the two or more images based on two or more images each including the same event and three-dimensional data of the structure; selecting a region of the event in each of two or more images; determining a location of the event in a coordinate system of the three-dimensional data based on the estimated camera position and orientation and a selected region in each of the two or more images; A method of image processing comprising: projecting three-dimensional data including the positions of the identified events to generate projected images in which the events are combined in a manner that makes them distinguishable from one another across the images.

[0063] [Appendix 11] estimating a position and an orientation of a camera at the time of capturing each of the two or more images based on two or more images each including the same event and three-dimensional data of the structure; selecting a region of the event in each of two or more images; determining a location of the event in a coordinate system of the three-dimensional data based on the estimated camera position and orientation and a selected region in each of the two or more images; A program for causing a computer to execute a process including projecting three-dimensional data including the positions of the identified events and generating projected images in which the events are synthesized in a manner that allows them to be distinguished from one another for each image.

[0064] Some or all of the elements (e.g., configurations and functions) described in Supplementary Notes 2 to 9 that are dependent on Supplementary Note 1 may also be dependent on Supplementary Notes 10 and 11 in the same dependent relationship as Supplementary Notes 2 to 9. Some or all of the elements described in any Supplementary Note may be applied to various hardware, software, recording means for recording software, systems, and methods. [Explanation of symbols]

[0065] 10: Image processing device 11: Posture estimation part 12: Area selection section 13: 3D position calculation section 14: Projection section 100: Image processing device 101: Image input unit 102: Camera pose estimation unit 103: Event domain selection section 104: Event 3D location identification unit 105: Event Projection Department 120: 3D data database 130: Event DB 500: Computer equipment 510: Processor 520: Storage section 530:ROM 540:RAM 550: Communication IF 560: User IF

Claims

1. a posture estimation unit that estimates a position and a posture of a camera at the time of capturing each of two or more images, each of which includes the same event, based on three-dimensional data of a structure; a region selection unit for selecting a region of the event in each of two or more images; a three-dimensional position specifying unit that specifies a position of the event in a coordinate system of the three-dimensional data based on the estimated camera position and orientation and a region selected in each of the two or more images; a projection unit that projects three-dimensional data including the positions of the identified events and generates projected images in which the events are synthesized in a manner that allows them to be distinguished from one another for each image.

2. The image processing device according to claim 1 , wherein the projection unit changes at least one of a transmittance to be set and a display color to be assigned in the projected image for each of the events.

3. The image processing device according to claim 1 , wherein the two or more images are captured at different times.

4. The image processing device according to claim 3 , wherein the projection unit shifts and overlays the positions of the events such that an image captured earlier is overlaid above or below the image in the projected image.

5. 3. The image processing device according to claim 1, wherein the projection unit determines a plane to which each event belongs as a projection plane based on the position of the event in the coordinate system of the identified three-dimensional data, and determines a projection range to be represented in the projection image on the determined projection plane.

6. 6. The image processing device according to claim 5, wherein the projection unit calculates a curvature of a surface including a predetermined range including the position of the identified event from normals of each data point of the three-dimensional data of the predetermined range, and determines, as the projection surface, a surface in the three-dimensional data that has the same curvature as the calculated curvature in an area continuous with the predetermined range.

7. The image processing device according to claim 1 , wherein the projection unit displays a scale in the projected image.

8. an event database for storing, for each of the events, information identifying an image in which the event is included, information on the area of ​​the event within the image, and information on the estimated position and orientation of the camera; the posture estimation unit acquires the information on the position and posture of the camera from the event database when the information on the position and posture of the camera is stored in the event database; The image processing device according to claim 1 , wherein the area selection unit acquires the information on the area of ​​the event from the event database when the information on the area of ​​the event is stored in the event database.

9. estimating a position and an orientation of a camera at the time of capturing each of the two or more images based on two or more images each including the same event and three-dimensional data of the structure; selecting a region of the event in each of two or more images; determining a location of the event in a coordinate system of the three-dimensional data based on the estimated camera position and orientation and a selected region in each of the two or more images; A method of image processing comprising: projecting three-dimensional data including the locations of the identified events to generate projected images in which the events are combined in a distinguishable manner from one image to another.

10. estimating a position and an orientation of a camera at the time of capturing each of the two or more images based on two or more images each including the same event and three-dimensional data of the structure; selecting a region of the event in each of two or more images; determining a location of the event in a coordinate system of the three-dimensional data based on the estimated camera position and orientation and a selected region in each of the two or more images; A program for causing a computer to execute a process including projecting three-dimensional data including the positions of the identified events, and generating projected images in which the events are synthesized in a manner that allows them to be distinguished from one another for each image.

Images