Image processing device, image processing method and image processing program

Abstract

One embodiment of the present invention provides an image processing device, an image processing method, and an image processing program that can easily process a three-dimensional model and damage information of a building based on an image obtained by photographing the building. In one embodiment, the image processing device includes: a processor; and a memory that stores a plurality of images obtained by photographing the building and a three-dimensional model of the building in which components constituting the building are determined, the memory stores the plurality of images and the components in association with each other, and the processor performs: extraction processing to extract damage information of the building based on the plurality of images; selection processing to select an image corresponding to a specified component from the plurality of images according to a specified criterion; and output processing to output the specified component, the selected image, and the damage information in association with each other.

Description

Technical Field

[0001] The present invention relates to a technology for processing images obtained by photographing buildings, three-dimensional models of buildings, and damage information. Background Art

[0002] Regarding the technology for processing a three-dimensional model and damage information of a building, for example, Patent Document 1 describes a device for creating a management map of a structure (building, structure) such as a bridge.

[0003] Previous technical literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Publication No. 2011-192270 Summary of the invention

[0006] Technical issues to be solved by the invention

[0007] One embodiment of the technology disclosed herein provides an image processing device, an image processing method, and an image processing program that can easily process an image obtained by photographing a building, a three-dimensional model of the building, and damage information.

[0008] Means for solving technical problems

[0009] The image processing device involved in the first embodiment of the present invention comprises: a processor; and a memory, which stores multiple images obtained by photographing a building and a three-dimensional model of the building in which the components constituting the building are determined, and the memory stores the multiple images and the components in association with each other, and the processor performs: extraction processing, extracting damage information of the building based on the multiple images; selection processing, selecting an image corresponding to a specified component from the multiple images according to a specified criterion; and output processing, outputting the specified component, the selected image, and the damage information in association with each other.

[0010] In the image processing device involved in the second method, in the first method, the processor performs: generation processing, generating a three-dimensional model of a building based on multiple images; determination processing, determining the components constituting the building in the generated three-dimensional model; and storage control processing, storing the three-dimensional model in a memory in association with the multiple images and the determined components.

[0011] The image processing device according to the third aspect is characterized in that, in the second aspect, the processor performs the specifying process independently of the operation of the specifying means by the user.

[0012] The image processing device according to a fourth aspect is characterized in that, in any one of the first to third aspects, the processor performs an acceptance process for accepting designation of an acceptance criterion.

[0013] In the image processing device according to a fifth aspect, in any one of the first to fourth aspects, the processor extracts at least one of the type, number, size, degree of damage, and time change of damage as the damage information in the extraction process.

[0014] The image processing device according to a sixth aspect is characterized in that, in any one of the first to fifth aspects, the processor selects the image for each lesion type in the selection process.

[0015] The image processing device according to a seventh aspect is characterized in that, in any one of the first to sixth aspects, the processor selects a designated number of images in the selection process.

[0016] The image processing device according to an eighth aspect is characterized in that, in any one of the first to seventh aspects, the processor performs image placement processing for placing the selected image in an area designated as an image area in a document in a designated format.

[0017] The image processing device according to a ninth aspect is configured such that, in the eighth aspect, the processor performs information input processing for inputting the damage information into an area designated as an information area in the document.

[0018] In the image processing device involved in the tenth embodiment, in any one of the first to ninth embodiments, the processor performs: a first display processing of displaying the three-dimensional model and position information representing the position of the selected image in the three-dimensional model on the display device in association with each other; and a second display processing of displaying the selected image on the display device with respect to the position information specified in the displayed position information.

[0019] In the image processing device involved in the eleventh method, in any one of the first to tenth methods, the processor performs: a first display processing, displaying the three-dimensional model and position information representing the position of the selected image in the three-dimensional model on the display device in association with each other; and a third display processing, displaying the selected image on the display device with respect to the displayed position information.

[0020] The image processing device according to the twelfth aspect is the image processing device of the tenth or eleventh aspect, wherein the processor recognizes the display position information in a manner corresponding to the damage information at least in the first display process.

[0021] In the image processing device involved in the thirteenth method, in any one of the tenth to twelfth methods, the processor synthesizes an image corresponding to a specific component from a plurality of images, maps the synthesized image onto the specific component, and displays it on the display device, at least in the first display processing.

[0022] The image processing device according to a fourteenth aspect is the image processing device according to any one of the tenth to thirteenth aspects, wherein the processor displays the damage information by emphasizing it on the three-dimensional model at least in the first display process.

[0023] In the image processing device involved in the fifteenth method, in any one of the first to fourteenth methods, the processor performs: acquisition processing, acquiring multiple images obtained by photographing a building, that is, multiple images with shooting dates and times different from multiple images stored in a memory; and association processing, associating the acquired multiple images with parts of a three-dimensional model stored in the memory.

[0024] The image processing device according to the sixteenth aspect is characterized in that, in the fifteenth aspect, the processor performs the association process based on the mutual relationship between the plurality of acquired images and the plurality of images stored in the memory.

[0025] The image processing method involved in the seventeenth embodiment of the present invention is an image processing method performed by an image processing device, the image processing device comprising: a processor; and a memory storing a plurality of images obtained by photographing a building and a three-dimensional model of the building, the memory storing the plurality of images in association with the components constituting the building in the three-dimensional model, the processing performed by the processor including: an extraction step of extracting damage information of the building based on the plurality of images; a selection step of selecting an image corresponding to a component specified in the three-dimensional model from the plurality of images according to a specified criterion; and an output step of outputting the specified component, the selected image, and the damage information in association with each other. The image processing method involved in the seventeenth embodiment may further have the same structure as the second to sixteenth embodiments.

[0026] The image processing program according to the eighteenth aspect of the present invention causes a computer to execute the image processing method according to the seventeenth aspect. A non-transitory recording medium recording a computer-readable code of the image processing program according to the eighteenth aspect can also be cited as an aspect of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a diagram showing a schematic configuration of an image processing system according to an embodiment.

[0028] Figure 2 It is a diagram showing the functional structure of the processing unit.

[0029] Figure 3 A diagram showing information stored in a storage device.

[0030] Figure 4 is a flowchart showing the processing of the image processing method.

[0031] Figure 5 This is a diagram showing a situation in which an image group is acquired.

[0032] Figure 6 The figure shows an example of three-dimensional point cloud data.

[0033] Figure 7 It is a figure which shows an example of a three-dimensional model.

[0034] Figure 8 It is a diagram showing an example of damage information.

[0035] Fig. 9 This is a diagram showing a situation in which a selection criterion for a representative image is set.

[0036] Fig.10 This is a diagram showing how the output format of the processing result is set.

[0037] Fig.11 This is a diagram showing a situation where a representative image and damage information are input into a two-dimensional inspection record.

[0038] Fig.12 This is a diagram showing how position information is displayed on a three-dimensional model.

[0039] Fig.13 This is a diagram showing a state where a representative image at a designated position is displayed.

[0040] Fig.14 This diagram shows a situation where a representative image is displayed from the beginning.

[0041] Fig.15 This is a diagram showing a state where a portion of a representative image including a designated position in a two-dimensional point inspection record is displayed.

[0042] Fig.16 This is a diagram showing a state where a synthesized image is mapped onto a three-dimensional model. DETAILED DESCRIPTION

[0043] An embodiment of an image processing device, an image processing method, and an image processing program according to the present invention is described below. In the description, the drawings are referred to as needed.

[0044] [Structure of image processing system]

[0045] Figure 1 This is a block diagram showing the schematic structure of an image processing system 1 (image processing device). The image processing system 1 includes an image processing device 10 (image processing device) and a display device 20 (display device, monitor), and is a system that extracts damage information from multiple images obtained by split-lens photography of a subject, creates a three-dimensional model, assists in the creation of two-dimensional inspection records, and the like. The image processing system 1 can be constructed using devices (information terminals) such as personal computers, tablet terminals, and smart phones. The various elements of the image processing system 1 can be housed in one frame or in separate frames. In addition, the various elements can also be configured at a remote location and connected via a network.

[0046] [Structure of Image Processing Device]

[0047] The image processing device 10 includes a processing unit 100 , a storage device 200 , and an operation unit 300 , and these units are connected to each other so as to transmit and receive necessary information.

[0048] [Structure of the processing unit]

[0049] Figure 2 1 is a diagram showing the structure of the processing unit 100 (processor). The processing unit 100 includes an input processing unit 102, an acquisition processing unit 103, an extraction processing unit 104, a generation processing unit 105, a determination processing unit 106, an association processing unit 107, a selection processing unit 108, a storage control processing unit 109, a reception processing unit 110, an image configuration processing unit 112, an information input unit 114, a display processing unit 116, an image synthesis unit 117, and a communication control unit 118, and performs acquisition of photographic images, creation of three-dimensional models, and assistance in creation of two-dimensional inspection records. The details of the processing performed by each of these units will be described later.

[0050] The functions of the processing unit 100 described above can be implemented using various processors and recording media. Various processors include, for example, a general-purpose processor that executes software (programs) to implement various functions, namely a CPU (Central Processing Unit), a processor dedicated to image processing, namely a GPU (Graphics Processing Unit), and a processor whose circuit structure can be changed after manufacturing, namely a programmable logic device (PLD).

[0051] Each function can be implemented by one processor, or by multiple processors of the same or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA, or a combination of a CPU and a GPU). In addition, multiple functions can be implemented by one processor. More specifically, the hardware structure of these various processors is a circuit composed of circuit elements such as semiconductor elements.

[0052] When the processor or circuit described above executes software (program), a computer-readable code of the executed software (e.g., various processors or circuits and / or a combination thereof constituting the processing unit 100) is stored in advance in a non-temporary recording medium (memory) such as a ROM, and the computer refers to the software. During execution, the information stored in the storage device is used as needed. In addition, during execution, for example, a RAM (Random Access Memory; memory) can be used as a temporary storage area.

[0053] Alternatively, a server on the network may implement part or all of the functions of the processing unit 100, and the image processing device 10 may perform data input, communication control, result display, etc. In this case, an Application Service Provider type system including a server on the network is constructed.

[0054] [Structure of storage unit]

[0055] The storage device 200 (storage device, memory) is composed of a non-temporary recording medium such as a CD (Compact Disk), a DVD (Digital Versatile Disk), a hard disk (Hard Disk), or various semiconductor memories, and a control unit thereof. Figure 3 The information shown is stored in association with each other. Photographic image 202 is a plurality of images obtained by photographing a building, and composite image 204 is (a collection of) images corresponding to certain components synthesized from photographic images. Three-dimensional model data 206 (three-dimensional model) is a three-dimensional model of a building created based on a photographic image, and determines the components constituting the building. In addition, three-dimensional model data 206 is associated with photographic images, representative images, two-dimensional records, etc., and as described in detail later, by the user specifying position information on the three-dimensional model, a representative image or a two-dimensional inspection record can be displayed. Damage information 208 (damage information) is information extracted from a photographic image that indicates damage to a building. Inspection record data 210 is a template (a document in a specified format) of a two-dimensional inspection record or data in which a representative image or damage information is configured and input into a template (described later). The template may also be in a format prescribed by the Ministry of Land, Infrastructure, Transport and Tourism or a local government.

[0056] In addition to these information, parameters of the imaging device (focal length, image sensor image size, pixel pitch, etc.) required when applying SfM (Structure from Motion) described later may also be stored in the storage device 200 .

[0057] [Structure of the operation unit]

[0058] The operation unit 300 includes a keyboard 310 and a mouse 320, and the user can perform operations required for the image processing according to the present invention through these devices. The display device 20 may be used as the operation unit by using a touch panel type device.

[0059] [Display device]

[0060] The display device 20 (display device) is, for example, a device such as a liquid crystal display, and can display information such as acquired photographic images, damage information, three-dimensional models, two-dimensional inspection records, and representative images.

[0061] [Image processing order]

[0062] Figure 4 This is a flowchart showing the procedure of the image processing method of the present invention.

[0063] [Image input]

[0064] The input processing unit 102 (processor) inputs a plurality of images obtained by photographing a building as a subject (step S100: input processing, input process). Buildings (buildings, structures) are, for example, bridges, roads, etc., but may also be other buildings. In addition, the input processing unit 102 may input an image stored in the storage device 200 as a photographic image 202, or may input an image via a recording medium or a network not shown. These images may be taken while moving the viewpoint by an aircraft such as a drone or a robot with a mobile function (or may be taken by a user). The photographed image may not be a stereoscopic image. In addition, in order to create a three-dimensional model and synthesize images, it is preferred that many feature points are common between images, and therefore it is preferred that adjacent images overlap each other sufficiently (for example, more than 80% of the area). Figure 5 FIG. 2 is a diagram showing a state where such superimposed images are captured by the imaging device 30 .

[0065] [Extraction of damage]

[0066] The extraction processing unit 104 (processor) extracts damage information of the building based on the input multiple images (step S110: extraction processing, extraction step). The extraction processing unit 104 can extract at least one of the type, number, size, degree of damage and time change of damage as damage information in the extraction processing.

[0067] The extraction processing unit 104 may extract damage information using various methods. For example, the crack detection method described in Japanese Patent No. 4006007 or the rust and peeling detection method described in Japanese Patent Table No. 2010-538258 may be used. In addition, the extraction processing unit 104 may extract damage information using a machine learning method. For example, an image with a type or size of damage as a label may be given as supervised data, and a learner such as DNN (Deep Neural Network) may be generated through machine learning, and damage may be detected using the generated learner.

[0068] The extraction processing unit 104 may extract damage information from each photographic image and synthesize the corresponding information into one, or may extract damage information from a single image synthesized from a plurality of photographic images. Damage may be represented as a vector having a start point and an end point, and in this case, the hierarchical structure of the vectors may also be considered as described in WO2017 / 110279.

[0069] Figure 8 2 is a diagram showing an example of the extracted damage information. The extraction processing unit 104 can store the extracted damage information in the storage device 200 as the damage information 208 .

[0070] [Creation of 3D model]

[0071] The generation processing unit 105 (processor) creates a three-dimensional model of the building based on the input multiple images (step S120: generation processing, generation process). In the three-dimensional model, there are various models such as a three-dimensional point cloud model, a three-dimensional surface model or a three-dimensional polygon model created based on the three-dimensional point cloud model, or a three-dimensional model with an image mapped on the texture. The generation processing unit 105 can use the SfM (Structure from Motion) method to create the three-dimensional model. SfM is a method of restoring a three-dimensional shape based on multi-viewpoint images. For example, feature points are calculated using algorithms such as SIFT (Scale-Invariant Feature Transform), and the feature points are used as clues to calculate the three-dimensional position of the point cloud using the principle of triangulation. Specifically, a straight line is drawn from the camera device to the feature point using the principle of triangulation, and the intersection of the two straight lines passing through the corresponding feature points becomes the restored three-dimensional point. Then, by performing this operation on each detected feature point, the three-dimensional position of the point cloud can be obtained. Figure 6 1 is a diagram showing a point cloud 500 (an example of a point cloud).

[0072] In SfM, although the size is not calculated, it is possible to associate the size with the actual size by, for example, placing a scaler of known size on the subject and photographing the subject.

[0073] The generation processing unit 105 can apply, for example, a TIN model (TIN: triangulated irregular network) to the data of the point cloud thus obtained and use triangles to approximate the surface of the building, and obtain a solid model (solid model; three-dimensional model) based on the result. In the solid model, the three-dimensional shape of the building is composed of a combination of three-dimensional components such as filled building blocks or blocks. In addition, when obtaining the solid model, the user may specify "which range of the point cloud belongs to the same surface" via the operation unit 300, and the generation processing unit 105 may use the result. In addition, the generation processing unit 105 may use algorithms such as RANSAC (Random Sample Consensus) to automatically generate the solid model without relying on the user's operation. When generating the solid model, the generation processing unit 105 may use the information of the three-dimensional position, color (R, G, B), and brightness of the point cloud to calculate the changes in this information.

[0074] [Use of generated 3D models]

[0075] In addition, when a three-dimensional model has already been generated or acquired through past inspections, etc., the generation processing unit 105 can also read in the model. When using such a generated three-dimensional model, the acquisition processing unit 103 can acquire multiple images obtained by photographing the building, that is, multiple images with different shooting dates and times from the multiple images stored in the storage device 200 (memory) (for example, images with a shooting date and time that is newer than the stored images and not used for the generation of the three-dimensional model) (acquisition processing, acquisition process). In addition, the association processing unit 107 can associate the multiple acquired images with the components of the three-dimensional model stored in the storage device 200 (association processing, association process). The association processing unit 107 can, for example, perform association processing (association process) based on the relationship between the multiple acquired images and the multiple images stored in the storage device 200.

[0076] [Determination of parts]

[0077] The determination processing unit 106 (processor) determines the components constituting the building in the three-dimensional model (step S130: determination processing, determination process). That is, the determination processing unit 106 determines "which component of the building corresponds to each area of ​​the three-dimensional model". The determination processing unit 106 can determine the component based on the user's operation, or it can determine it without relying on the user's operation to determine the component. The determination processing unit 106 can also use information related to the shape or size of the component when determining the component. For example, information such as "the component that is two-dimensionally expanded in the horizontal plane and has an area greater than a threshold is the base plate" and "the component that is attached to the base plate and extends one-dimensionally is the main beam" can be used. In addition, the determination processing unit 106 can also use a learner such as a DNN constructed by machine learning that gives the components constituting the three-dimensional model as the correct label to determine the component. Figure 7 1 is a diagram showing an example of a three-dimensional model in which components are determined. In the example of this figure, a three-dimensional model 510 of a bridge is composed of components such as a bottom plate 512 , a wall 514 , and a leg 516 .

[0078] The storage control processing unit 109 can store the data representing the generated three-dimensional model in the storage device 200 as the three-dimensional model data 206 (storage control processing, storage control step).

[0079] [Selection of representative images]

[0080] The above-mentioned point cloud is information indicating the position of the feature points contained in the photographic image. In addition, since the components are determined in step S130, the photographic images corresponding to each component can be determined. However, since a large number of images are usually acquired for each component in order to create a three-dimensional model, it is a time-consuming and laborious task for the user to select images. Therefore, in the image processing system 1, the acceptance processing unit 110 (processor) accepts the designation of the image selection criterion (step S140: acceptance processing, acceptance process), and the selection processing unit 108 (processor) selects the image (representative image) corresponding to the determined component from the captured image according to the criterion (step S140: selection processing, selection process).

[0081] Fig. 9 2 is a diagram showing a situation where a selection criterion for a representative image is set, and the reception processing unit 110 displays such a screen on the display device 20. Fig. 9 In the example, the receiving processing unit 110 sets a priority for the information contained in the damage information (the number of damages, size, degree (level) of damages, and time changes) based on the user's operation. In addition, the receiving processing unit 110 sets the number of representative images to be selected (it can be one or more) based on the user's operation. Based on such a setting, the selection processing unit 108 selects a specified number of images according to the set priority. As a result, the user can quickly and easily select representative images. In addition, Fig. 9 The illustrated method is an example of setting the selection criteria, and the representative image may be selected based on other criteria (for example, the image quality of the photographed image, the type of component, the identification number of the component, etc.).

[0082] [Output of processing results]

[0083] Fig.10 FIG. 1 is a diagram showing how the output method of the processing result is set. Fig.10 In the example shown, the user can select the display of the two-dimensional inspection record, the display of the three-dimensional model, the display timing of the representative image, the presence or absence of mapping of the synthetic image, the emphasized display of the damage, and the presence or absence of display of the two-dimensional inspection record on the three-dimensional model by selecting the radio button and inputting the numerical value through the operation unit 300. Then, according to the set content, the image configuration processing unit 112, the information input unit 114, and the display processing unit 116 independently or collaboratively output the determined parts, the selected images, and the damage information in association (step S150: output processing, output process). The processing of steps S110 to S150 can be repeated until the judgment of ending in step S160 is affirmed. In addition, the setting of the output method and the output in the set method can be performed at the time desired by the user. For example, after the processing results are displayed in a certain mode, they can be displayed again in another mode. The specific content of each output method is described below.

[0084] [Part 1: Output using 2D inspection records]

[0085] Fig.11 This is an example in which the image arrangement processing unit 112 arranges representative images (representative images 552, 554, 556, 558) in an area (image area) designated as an area for arranging images in a two-dimensional spot check record (an example of a "document in a specified format"). In this figure, a float 552A is captured in the representative image 552, and a peeling 554A is captured in the representative image 554. This form of record can also span multiple pages. Fig.11 In the example, the information input unit 114 also inputs damage information such as the photo number, component name, element number, type and degree of damage into the area designated as the information area in the document (information input processing, information input process). The content of the "Remarks" column can be automatically input by the information input unit 114 based on the damage information, or can be input based on the user's operation. Fig.10 When the “two-dimensional inspection record display mode” is turned on in the screen of FIG. 1 , the display processing unit 116 displays such a screen on the display device 20 . In addition, the display processing unit 116 stores information corresponding to the screen as the inspection record data 210 .

[0086] In addition, the formats of various documents or the layout of representative images can be considered. For example, documents such as inspection records can be in a format determined by the Ministry of Land, Infrastructure, Transport and Tourism, local governments, etc., or in other formats. Documents in other specified formats other than inspection records can also be used. In addition, representative images can be classified and arranged in documents using components as keywords, and representative images of each component can be classified and arranged using the type of damage as a keyword (for each type of damage).

[0087] By such processing, the user can quickly and easily select a representative image and create an inspection record in the image processing system 1. That is, the user can easily process an image of a building, a three-dimensional model of the building, and damage information.

[0088] [Second: Using the output of the 3D model]

[0089] Fig.12 This is a diagram showing a state (result of the first display process) where a three-dimensional model 520 and pins 532 and 534 (position information) indicating the position of a selected image (representative image) in the three-dimensional model 520 are displayed on the display device in association with each other. Fig.10 In the case where the display timing of the representative image is "when the needle is specified", the needle is displayed at the position of the representative image. Fig.12In the example, the three-dimensional model is a model showing the wall near the leg of the bridge, and the display processing unit 116 can enlarge or reduce the three-dimensional model according to the user's operation via the operation unit 300, and also move the viewpoint or change the line of sight. In addition, the display processing unit 116 can also turn on or off the display of a specific part (for example, a part indicated by the user) when displaying the three-dimensional model. Fig.10 When the "three-dimensional model display mode" is turned on, the display processing unit 116 performs such a display. Figures 12 to 15 A 3D model without texture is shown in FIG. 1 , but the same processing can be performed on a 3D model with texture (see Fig.16 and records related to the diagram).

[0090] The display processing unit 116 may also highlight the damage information on the three-dimensional model. For example, the damage such as cracks, lifting, peeling, or the like or their outlines may be displayed by tracing and displaying them with thick lines or lines of conspicuous colors.

[0091] In addition, Fig.12 In the example of , the display processing unit 116 may also identify and display the needle in a manner corresponding to the damage information. Specifically, the display processing unit 116 changes the mark of the needle according to the type of damage for display, but in addition to the mark, the color or number may be changed, or it may be displayed as an icon. In addition, the display method may be changed according to the degree of damage. Fig.12 In the example, for example, the user can select (click, etc.) the markers 532 and 534 by moving the cursor 540 on the screen using the mouse 320.

[0092] Fig.13 The diagram shows a state (result of the second display process) in which the display processing unit 116 displays a representative image on the display device 20 for a designated pin among the pins (position information) displayed on the screen. Fig.13 Part (a) of the figure shows a state where a representative image 552 is "floating" as designated by a needle 532, and part (b) of the figure shows a state where a representative image 554 is "peeled off" as designated by a needle 534. The display processing unit 116 may display the representative image in a region different from the three-dimensional model, on a different screen, or on a different display device.

[0093] Fig.14 1 is a diagram showing a state where a representative image is displayed on a three-dimensional model from the beginning (the result of the first display process and the third display process). Fig.10 When the display timing of the representative image in is set to "from the beginning", the display processing unit 116 performs such display. Fig.14In the example shown, the pins 532, 534 and the representative images 552, 554 are connected by a lead line, so that the user can easily understand the relationship between the pins and the representative images. The representative images may also be displayed on balloon-shaped graphics connected to the pins.

[0094] Fig.15 FIG. 4 is a diagram showing a state where a portion of a two-dimensional inspection record including a representative image corresponding to a selected pin is displayed (result of the fourth display process). Fig.10 When "Display 2D inspection record on 3D model" is turned on, the display processing unit 116 can perform such display. The display processing unit 116 can also display such partial 2D inspection record in a different area, different screen, or different display device from the 3D model.

[0095] Through such processing, the user can quickly and easily browse the representative image or the created inspection record in the image processing system 1. That is, the user can easily process the image obtained by photographing the building, the three-dimensional model of the building, and the damage information.

[0096] [Part 3: Mapping of synthetic images to 3D models]

[0097] Fig.16 The display device 20 displays the composite image mapped onto the three-dimensional model 520 (see Fig.12 ) is a diagram showing the situation of the three-dimensional model 522 on the Fig.10 This display is performed when "synthesized image mapping" is turned on in the display. In this mode, the image synthesis unit 117 (processor) synthesizes images corresponding to the specified components in multiple photographic images, and the display processing unit 116 maps the synthesized image to the specified component and displays it on the display device 20 (fifth display processing). Image synthesis and mapping can be performed on a part of the building or on all the components. In addition, in Fig.16 In the example shown, the display processing unit 116 performs identification display of the position information in the same manner as described above (displaying the needles 532, 534, 536 with marks corresponding to the types of damage), and may also display a representative image or a two-dimensional inspection record based on the selection of the position information. In addition, the damage information may be highlighted.

[0098] In addition, Fig.16, an example of mapping a separately generated synthetic image only onto a three-dimensional model of a three-dimensional shape is shown in FIG. However, the generation processing unit 105 (processor) may also directly generate a textured three-dimensional model by using the above-mentioned SfM or the like. In this case, the generation processing unit 105 may also use other algorithms such as MVS (Multi-View Stereo). MVS can generate a more detailed three-dimensional model because it contains more information than monocular. For this textured three-dimensional model, it can also be compared with the Figures 12 to 15 The above-described method also displays position information or representative images, displays a portion of a two-dimensional inspection record, and emphasizes the display of damage information.

[0099] By performing such processing, the user can easily process an image of a building, a three-dimensional model of the building, and damage information in the image processing system 1 .

[0100] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present invention.

[0101] Explanation of symbols

[0102] 1 Image Processing System

[0103] 10 Image processing device

[0104] 20 Display device

[0105] 30 Camera device

[0106] 100 Processing Department

[0107] 102 Input processing unit

[0108] 103 Acquisition Processing Unit

[0109] 104 Extraction Processing Unit

[0110] 105 Generation Processing Unit

[0111] 106 Determine the processing department

[0112] 107 Association Processing Department

[0113] 108 Select Processing Unit

[0114] 109 Storage Control Processing Unit

[0115] 110 Receiving and Processing Department

[0116] 112 Image configuration processing unit

[0117] 114 Information input unit

[0118] 116 Display Processing Unit

[0119] 117 Image Synthesis Unit

[0120] 118 Communication Control Department

[0121] 200 Storage Device

[0122] 202 Photographic images

[0123] 204 Composite Image

[0124] 206 3D model data

[0125] 208 Damage Information

[0126] 210 Inspection record data

[0127] 300 Operation Department

[0128] 3110 Keyboard

[0129] 320 Mouse

[0130] 500 point clouds

[0131] 510 3D Model

[0132] 512 Base Plate

[0133] 514 Wall

[0134] 516 Legs

[0135] 520 3D models

[0136] 522 3D Models

[0137] 532 Pin

[0138] 534 Pin

[0139] 536 Pin

[0140] 540 Cursor

[0141] 552 represents image

[0142] 552A Float

[0143] 554 represents image

[0144] 554A Stripping

[0145] 556 represents image

[0146] 558 represents image

[0147] S100~S160 Steps of the image processing method

Claims

1. An image processing device comprising: Processor; and a memory storing a plurality of images obtained by photographing a building and a three-dimensional model of the building in which components constituting the building are determined, wherein the memory stores the plurality of images and the components in association with each other, The processor performs: Extraction processing, extracting damage information of the building based on the multiple images; A selection process of selecting an image corresponding to a designated component from the plurality of images according to a designated criterion; as well as output processing, outputting the designated component, the selected image, and the damage information in association with each other, The specified criteria include the priority of the information items included in the damage information or the image quality of the photographic image set according to the user's operation, The processor selects a designated number of images according to the designated criterion in the selection process.

2. The image processing device according to claim 1, wherein: The processor performs: A generation process of generating a three-dimensional model of the building based on the multiple images; Determining processing, determining components constituting the building in the generated three-dimensional model; as well as The storage control process stores the three-dimensional model in the memory in association with the plurality of images and the determined components.

3. The image processing device according to claim 2, wherein: The processor performs the determination processing independently of a user determining the operation of the component.

4. The image processing device according to any one of claims 1 to 3, wherein: The processor performs an acceptance process of accepting a designation of the reference.

5. The image processing device according to any one of claims 1 to 3, wherein: The processor extracts at least one of the type, number, size, degree of damage, and time change of damage as the damage information in the extraction process.

6. The image processing device according to any one of claims 1 to 3, wherein: The processor selects an image for each type of lesion in the selection process.

7. The image processing device according to any one of claims 1 to 3, wherein: The processor performs image arrangement processing of arranging the selected image in an area designated as an image area in a document in a designated format.

8. The image processing device according to claim 7, wherein: The processor performs information input processing of inputting the damage information into an area designated as an information area in the document.

9. The image processing device according to any one of claims 1 to 3, wherein: The processor performs: a first display process of displaying the three-dimensional model and position information indicating the position of the selected image in the three-dimensional model on a display device in association with each other; as well as The second display processing causes the selected image to be displayed on the display device at the position information designated in the displayed position information.

10. The image processing device according to any one of claims 1 to 3, wherein: The processor performs: a first display process of displaying the three-dimensional model and position information indicating the position of the selected image in the three-dimensional model on a display device in association with each other; as well as The third display processing is to display the selected image on the display device with respect to the display position information.

11. The image processing device according to claim 9, wherein: The processor recognizes and displays the position information in a manner corresponding to the damage information at least in the first display process.

12. The image processing apparatus according to claim 9, wherein: The processor synthesizes an image corresponding to the determined component among the plurality of images, maps the synthesized image onto the determined component, and displays the synthesized image on the display device at least in the first display process.

13. The image processing apparatus according to claim 9, wherein: The processor displays the damage information emphatically on the three-dimensional model at least in the first display process.

14. The image processing device according to any one of claims 1 to 3, wherein: The processor performs: Acquisition processing, acquiring a plurality of images obtained by photographing the building, that is, a plurality of images photographed on a different date and time from the plurality of images stored in the memory; as well as The process of associating the plurality of images acquired is performed to associate the components of the three-dimensional model stored in the memory.

15. The image processing apparatus according to claim 14, wherein: The processor performs the association process based on the mutual relationship between the plurality of images acquired and the plurality of images stored in the memory.

16. An image processing method, which is an image processing method executed by an image processing device, the image processing device comprising: Processor; and a memory storing a plurality of images obtained by photographing a building and a three-dimensional model of the building, wherein the memory stores the plurality of images in association with components constituting the building in the three-dimensional model, The processing performed by the processor includes: An extraction step of extracting damage information of the building based on the multiple images; A selection step of selecting, according to a specified criterion, an image corresponding to a component specified in the three-dimensional model from the plurality of images; as well as an output step of outputting the designated component, the selected image, and the damage information in association with each other; The specified criteria include the priority of the information items included in the damage information or the image quality of the photographic image set according to the user's operation, The processor selects a designated number of images according to the designated criterion in the selection step. 17 . A computer-readable recording medium having recorded thereon an image processing program, the image processing program causing a computer to execute the image processing method according to claim 16 .

Images