Maintenance assistance device, maintenance assistance method, and recording medium

Abstract

The invention provides a maintenance assistance device, a maintenance assistance method, and a recording medium, which can well output the damage detection result when two or more kinds of damage are detected from a structure, especially two or more kinds of damage are detected from the same or close position of the structure. A maintenance assistance device is provided with a processor that performs: a damage detection process in which damage to a structure to be maintained is detected on the basis of an image acquired by capturing an image of the structure; a determination process for determining whether or not two or more types of damage are detected from the same or close position; and an output process for outputting damage detection results in a priority order of damage types when it is determined by the determination process that two or more types of damage are detected from the same or close position.

Description

[0001] This application is a divisional application of a Chinese patent application with an application date of March 1, 2021, an application number of 202180022542.7, and an invention title of "Electronic Device, Stop Determination Method, and Storage Medium". Technical Field

[0002] The present invention relates to an inspection assistance device, method, and program, and particularly to a technique for assisting the inspection of structures. Background Art

[0003] Social infrastructure structures such as bridges require regular inspections for maintenance and repair.

[0004] In Patent Document 1, a crack detection method and its display method are disclosed. In this method, the inner wall surface of a tunnel is photographed with a camera, and the photographed image is subjected to image processing, thereby extracting / quantifying cracks for each section of the inner wall surface and displaying crack information for each section. For example, according to the degree of cracks in each section, the cracks are displayed by differentiating colors for each section, whereby it is easy to grasp the degree of cracks.

[0005] Prior Art Documents

[0006] Patent Documents

[0007] Patent Document 1: Japanese Patent Application Laid-Open No. 2002-188998 Summary of the Invention

[0008] Technical Problem to be Solved by the Invention

[0009] However, there are many types of damages in structures other than cracks. For example, in the concrete part of a structure, in addition to cracks, there are multiple damages such as water leakage, free lime, rust juice, peeling, and steel bar exposure. Similarly, in the steel components of a structure, there are multiple damages such as cracks, corrosion, and deterioration of the anti-corrosion function.

[0010] In Patent Document 1, it is described that cracks are displayed by differentiating colors for each section according to the degree of cracks in each section, but it does not describe detecting two or more (multiple) damages from a structure, nor does it describe an output method for the detection results of multiple damages.

[0011] The present invention has been completed in view of such circumstances, and an object thereof is to provide an inspection assistance device, method, and program that can favorably output the damage detection results when two or more damages are detected from a structure, particularly when two or more damages are detected from the same or adjacent positions of a structure.

[0012] Means for Solving the Technical Problem

[0013] In order to achieve the above object, the invention according to the first aspect is a maintenance assistance device, which includes a processor that performs the following processes: an image acquisition process of acquiring an image of a structure to be maintained; a damage detection process of detecting damage to the structure based on the acquired image; a determination process of determining whether two or more damages are detected from the same or adjacent positions among the two or more damages when two or more damages to the structure are detected through the damage detection process; and an output process of outputting a damage detection result detected through the damage detection process, and when it is determined through the determination process that two or more damages are detected from the same or adjacent positions, outputting the damage detection result in the order of priority of damage types.

[0014] According to the first aspect of the present invention, when two or more damages to the structure are detected based on an image of the structure to be maintained, especially when two or more damages are detected from the same or adjacent positions of the structure, the damage detection result is output in the order of priority as the damage detection result. Thus, when two or more damages are detected from the same or adjacent positions of the structure, by outputting the damage detection result in the order of priority of damage types, it is possible to handle the situation where two or more damages are detected from the same or adjacent positions of the structure. In addition, when two or more damages are not at the same or adjacent positions of the structure, the two or more damage detection results can be directly output.

[0015] In the maintenance assistance device according to the second aspect of the present invention, the following is preferred: in the damage detection process, a damage area and the type of damage for each damage area are detected based on the image, the determination process determines whether two or more types of damage are detected in the same or adjacent damage areas through the damage detection process, and in the output process, when it is determined through the determination process that two or more types of damage are detected in the same or adjacent damage areas, the damage detection result of the damage type with the highest priority is output as the damage detection result for the same or adjacent damage areas. This is because when two or more damages are detected from the same or adjacent positions of the structure, it is effective to output the damage detection result of the damage type with the highest priority to notify the user or the like.

[0016] In the maintenance assistance device according to the third aspect of the present invention, the following is preferred: the adjacent position is a position where the distance between two or more damages is equal to or less than a threshold value.

[0017] In the maintenance assistance device according to the fourth aspect of the present invention, the following is preferred: in the damage detection process, when an image is input, a learned model that outputs a damage area and the type of damage for each damage area as an identification result is executed.

[0018] In the inspection assistance device according to the fifth aspect of the present invention, the following is preferred: In the output process, different drawing patterns are output in the case where the damage type is a linear damage and in the case where the damage type is a planar damage.

[0019] In the inspection assistance device according to the sixth aspect of the present invention, the following is preferred: In the output process, in the case where the damage type is a linear damage, a damage diagram representing a non-closed linear line is output, and in the case where the damage type is a planar damage, a damage diagram representing a closed line surrounding the planar damage is output. When outputting the damage diagram in a line drawing pattern, in the case of linear damage, it is set as a damage diagram representing a non-closed linear line, and in the case where the damage type is a planar damage, it is set as a damage diagram representing a closed line surrounding the planar damage.

[0020] In the inspection assistance device according to the seventh aspect of the present invention, the following is preferred: In the output process, in the case where the damage type is a linear damage, a damage image that at least fills the linear damage is output, and in the case where the damage type is a planar damage, a damage image that at least fills the planar damage is output.

[0021] In the inspection assistance device according to the eighth aspect of the present invention, the following is preferred: The output process outputs and displays the damage detection result on a display, or saves the damage detection result in a memory in the form of a file.

[0022] In the inspection assistance device according to the ninth aspect of the present invention, the following is preferred: The priority order of the damage types is a priority order preset according to the severity of the damage.

[0023] In the inspection assistance device according to the tenth aspect of the present invention, as the damage type, in the case of a linear damage including linear free lime and cracks, the priority order of the linear free lime is higher than that of the cracks.

[0024] In the inspection assistance device according to the eleventh aspect of the present invention, as the damage type, in the case of a planar damage including steel bar exposure, peeling, rust juice, planar free lime, and water leakage, it is set that the priority order becomes lower in the order of steel bar exposure, peeling, rust juice, planar free lime, and water leakage.

[0025] In the inspection assistance device according to the twelfth aspect of the present invention, the processor performs a priority acceptance process of accepting the priority order of the damage type of the structure from the operation unit operated by the user, and the priority order of the damage type is the priority order accepted from the user via the operation unit.

[0026] In the inspection assistance device according to the 13th aspect of the present invention, it is preferable that the processor performs the following processing: an edit instruction reception process of receiving an edit instruction for a damage detection result from an operation unit operated by a user; and an edit process of editing the damage detection result according to the received edit instruction.

[0027] In the inspection assistance device according to the 14th aspect of the present invention, it is preferable that the damage detection result includes a damage quantity table having items of damage identification information, damage type, and size, and information corresponding to each item is recorded for each detected damage.

[0028] The invention according to the 15th aspect is an inspection assistance method in which a processor performs inspection assistance for a structure to be inspected. Each process of the processor includes the following steps: acquiring an image of the structure to be inspected; detecting two or more types of damages of the structure based on the acquired image; determining whether two or more types of damages are detected from the same or adjacent positions among the two or more types of damages detected in the structure; and outputting the detected damage detection result, and when it is determined in the determination step that two or more types of damages are detected from the same or adjacent positions, outputting the damage detection result in the priority order of damage types.

[0029] The invention according to the 16th aspect is an inspection assistance program that causes a computer to execute a method for performing inspection assistance for a structure to be inspected. The method includes the following steps: acquiring an image of the structure to be inspected; detecting two or more types of damages of the structure based on the acquired image; determining whether two or more types of damages are detected from the same or adjacent positions among the two or more types of damages detected in the structure; and outputting the detected damage detection result, and when it is determined in the determination step that two or more types of damages are detected from the same or adjacent positions, outputting the damage detection result in the priority order of damage types.

[0030] Advantageous Effects of the Invention

[0031] According to the present invention, when two or more types of damages are detected from a structure, particularly when two or more types of damages are detected from the same or adjacent positions of the structure, the damage detection result can be output well. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a diagram showing an example of damage to a structure.

[0033] Figure 2 It is a diagram showing an example of linear free lime.

[0034] Figure 3 It is a diagram showing an example of planar free lime.

[0035] Figure 4 is a diagram showing the types of damage manifestation methods corresponding to the types of damage, Figure 4 where (A) of is a diagram showing an image including a crack, Figure 4 and (B) of is a diagram showing an image depicting a broken line along the crack.

[0036] Figure 5 is a diagram showing the types of damage manifestation methods corresponding to the types of damage, Figure 5 where (A) of is a diagram showing an image including delamination and exposed reinforcement, Figure 5 and (B) of is a diagram showing an image depicting a polygon surrounding the area of delamination and exposed reinforcement.

[0037] Figure 6 is a diagram showing the types of damage manifestation methods corresponding to the types of damage, Figure 6 where (A) of is a diagram showing an image including planar free lime, Figure 6 and (B) of is a diagram showing an image depicting a polygon surrounding the area of planar free lime.

[0038] Figure 7 is a diagram showing each broken line when cracks and linear free lime as linear damages are detected.

[0039] Figure 8 is for explaining Figure 7 the proximity judgment of cracks and linear free lime as linear damages shown in .

[0040] Figure 9 is a block diagram showing an example of the hardware structure of the maintenance assistance device according to the present invention.

[0041] Figure 10 is a conceptual diagram showing an embodiment of a damage detection processing unit composed of a CPU etc.

[0042] Figure 11 is a perspective view showing an example of a bridge to be maintained.

[0043] Figure 12 is a diagram showing an example of an ortho image corresponding to a coffer which is one of the maintenance units of the bridge.

[0044] Figure 13 is for showing Figure 12 an example of a damage detection result detected from the ortho image shown in .

[0045] Figure 14 is a diagram showing an example of an ortho image with a damage diagram corresponding to the coffer superimposed.

[0046] Figure 15 It is a diagram showing an example of a table of the number of damages included in the damage detection result.

[0047] Figure 16 It is a schematic diagram showing an example of the damage detection result of cracks and linear free lime based on the damage detection processing unit and its output processing.

[0048] Figure 17 It is a schematic diagram showing another example of the damage detection result of cracks and linear free lime based on the damage detection processing unit and its output processing.

[0049] Figure 18 It is a schematic diagram showing an example of the damage detection result of planar free lime and linear free lime based on the damage detection processing unit and its output processing.

[0050] Figure 19 It is a schematic diagram showing another example of the damage detection result of planar free lime and linear free lime based on the damage detection processing unit and its output processing.

[0051] Figure 20 It is a schematic diagram showing an example of the damage detection result of rust juice, planar free lime and water leakage based on the damage detection processing unit and its output processing.

[0052] Figure 21 It is a schematic diagram showing another example of the damage detection result of rust juice, planar free lime and water leakage based on the damage detection processing unit and its output processing.

[0053] Figure 22 It is a schematic diagram showing the GUI of the second embodiment of the damage detection result output, and is a diagram showing an example of the screen displayed on the display unit.

[0054] Figure 23 It is a schematic diagram showing the GUI of the second embodiment of the damage detection result output, and is a diagram showing another example of the screen displayed on the display unit.

[0055] Figure 24 It is a schematic diagram showing the GUI of the third embodiment of the damage detection result output. Figure 24 In (A) of [], it is a diagram showing the case where the transparency of the filling color as the damage image is set to "10". Figure 24 In (B) of [], it is a diagram showing a composite image in which a damage image with a transparency of "10" is overlapped and displayed on the image of the structure taken.

[0056] Figure 25 It is a schematic diagram showing the GUI of the third embodiment of the damage detection result output. Figure 25Figure (A) shows a case where the transparency of the filling color of the damage image is set to "50". Figure 25 Figure (B) shows a composite image formed by overlapping and displaying a damage image with a transparency of "50" on an image of a structure.

[0057] Figure 26 It is a schematic diagram of a GUI showing the third embodiment of the damage detection result output. Figure 26 Figure (A) shows a case where the transparency of the filling color of the damage image is set to "100". Figure 26 Figure (B) shows a composite image formed by overlapping and displaying a damage image with a transparency of "100" on an image of a structure.

[0058] Figure 27 It is a figure showing a method of adding vertices to a polygon surrounding a damage area.

[0059] Figure 28 It is a figure showing a method of deleting vertices from a polygon surrounding a damage area.

[0060] Figure 29 It is a flowchart showing an embodiment of the maintenance assistance method according to the present invention.

[0061] Symbol Explanation

[0062] 1 - Bridge, 2 - Main girder, 3 - Cross beam, 4 - Cross bracing, 5 - Cross strut frame, 6 - Bridge deck, 7 - Pier, 10 - Maintenance assistance device, 12 - Image acquisition unit, 13 - Image, 14 - Image database, 16 - Storage unit, 18 - Operation unit, 20 - CPU, 21 - Damage detection processing unit, 21A to 21C - Learned models, 22 - RAM, 24 - ROM, 26 - Display control unit, 27A to 27C - Damage detection results, 30 - Display unit, 40 - Screen, 42 - Check box, 44 - Setting screen, 45A - Slide button, 45B - Dialog box, A - Water leakage, B, C1 to C5 - Cracks, C1 - Free lime (planar), C2 - Free lime (linear), D - Rust juice, E - Delamination, F - Reinforcement exposure, H1 - Delamination, L1 to L6 - Shortest distances, P1 to P6 - Points of interest, S10 to S22 - Steps, X, Y - Polyline. Detailed Embodiment

[0063] Hereinafter, preferred embodiments of the maintenance assistance device, method, and program according to the present invention will be described with reference to the accompanying drawings.

[0064] [Summary of the Present Invention]

[0065] Figure 1It is a diagram showing an example of damage to a structure, particularly damage to concrete components that make up the structure.

[0066] Figure 1 In (A), it shows water leakage A, which is one of the phenomena caused by damage to the concrete component. Water leakage A is caused by water leaking from the damaged part due to damage to the concrete component (such as cracks, cracks in joints, poor joint fillers, etc.).

[0067] Figure 1 In (B), it shows crack B, free lime C1, and rust juice D generated in the concrete component. Free lime C1 refers to the phenomenon where it flows out from inside the concrete component due to water leakage, etc., and the lime component appears on the surface when the water evaporates. And rust juice D refers to the substance where steel materials such as steel bars inside the concrete component corrode, and brown corrosion products seep out to the concrete surface.

[0068] Figure 1 In (C), it shows peeling E and steel bar exposure F generated in the concrete component. Peeling E refers to the state where concrete flakes in a floating state peel off, and steel bar exposure F refers to the state where the steel bars inside the concrete are exposed as a result of peeling E.

[0069] Moreover, although not shown in the figure, as damage to steel components that make up the structure, there are types of damage such as cracks, corrosion, fracture, and deterioration of the anticorrosion function.

[0070] Figure 2 It is a diagram showing an example of linear free lime. Figure 3 It is a diagram showing an example of planar free lime.

[0071] Figure 2 The linear free lime C2 shown in (A) and (B) is the state where lime components are blocked in the cracks generated in the concrete component. Therefore, the linear free lime C2 and the cracks have substantially the same shape, and cracks are generated at the same position (area) as the linear free lime C2 on the linear free lime C2.

[0072] Figure 3 The planar free lime shown in (A) of expands downward along the crack with water leakage from the crack extending in the horizontal direction. Figure 3 The planar free lime C1 shown in (B) of expands centering on the crack in the joint. Figure 3 The planar free lime C1 shown in (C) of expands centering on the concrete crack.

[0073] One aspect of the present invention is to detect damage to a structure by taking an image of the structure to be inspected and output a damage detection result according to the type of detected damage.

[0074] Figures 4 to 6 It is a diagram showing the types of ways to represent damage corresponding to the types of damage.

[0075] As Figure 4 shown, in the case where a crack is detected ( Figure 4 in (A) of Figure 4 ), as a way to represent the crack, it is represented by a drawing pattern formed by an unclosed line (broken line) along the crack (

[0076] in (B) of Figure 2 ). This is because, in the case of a linear damage such as a crack, quantification of the length is required.

[0077] As Figure 5 shown, in the case where linear free lime is detected ( Figure 5 in (A) of Figure 5 ), as a way to represent the linear free lime, it is also output as a drawing pattern of a broken line along the linear free lime. And both the crack and the linear free lime are linear damages, but due to the different types of damage, it is preferable to represent them recognizably with broken lines of different line types (for example, colors).

[0078] As Figure 6 shown, in the case where delamination E and steel bar exposure F are detected ( Figure 6 in (A) of Figure 6 ), as a way to represent the delamination E and the steel bar exposure F, it is represented by a drawing pattern formed by a closed line (polygon) surrounding the planar damage area (

[0079] in (B) of

[0080] This is because, in the case of planar damages such as delamination E, quantification of the area is required.

[0081] As Figure 6 shown, in the case where planar free lime C1 is detected ( Figure 6 in (A) of Figure 6 ), as a way to represent the planar free lime C1, it is represented by a drawing pattern formed by a closed line (polygon) surrounding the planar damage area ( Figure 6 in (B) of Figure 6 ).

[0079] In addition, the planar free lime C1, the delamination E, and the steel bar exposure F are all planar damages, but due to the different types of damage, it is preferable to represent them recognizably with polygons of different line types (for example, colors).

[0080] In one embodiment of the present invention, even if classified as the same damage, when the shape of the damage is different, the type of damage is different, and it is represented by different drawing patterns according to its shape. For example, linear free lime is represented by a broken line, and planar free lime is represented by a polygon.

[0081] Moreover, on a structure (image), two or more types of damages may sometimes be detected from the same or nearby positions. Here, the nearby position means a position where the distance between two or more types of damages is below a threshold value. The threshold value can be determined by default or set by the user.

[0082] For example, in the case of linear free lime, the linear free lime overlaps with a concrete crack, and two damages are detected at the same position. And, as Figure 5 shown, in the case of delamination E and steel bar exposure F, the steel bar exposure F exists within the delamination E. Therefore, the distance between the two damages of the steel bar exposure F and the delamination E is below the threshold value, and both are damages at nearby positions.

[0083] Figure 7 is a diagram showing each broken line when detecting a crack and linear free lime as linear damages.

[0084] In Figure 7 , X is a broken line representing a crack, and Y is a broken line representing linear free lime.

[0085] Figure 8 is a diagram for explaining the proximity determination of a crack and linear free lime shown as linear damages in Figure 7 .

[0086] Hereinafter, with reference to Figure 8 , a determination example of "proximity" of two damages of a crack and linear free lime will be described.

[0087] As Figure 8 shown in (A) thereof, the shortest distance between the focus point P1 (the first vertex as the end point of the broken line X) of the broken line X representing the crack and the broken line of the linear free lime is set as L1.

[0088] Similarly, as Figure 8 shown in (B) thereof, the shortest distance between the focus point P2 (the second vertex) of the broken line X and the broken line Y is set as L2, and as Figure 8 shown in (C) thereof, the shortest distance between the focus point P3 (the third vertex) of the broken line X and the broken line Y is set as L3.

[0089] Moreover, as Figure 8 shown in (D) to (F) thereof, the respective shortest distances between the two broken lines X and Y are set as L4, L5, and L6. Additionally, the shortest distance L4 is the shortest distance of the focus point 4 (the vertex as the end point of the broken line Y) of the broken line Y, the shortest distance L5 is the shortest distance of the focus point 5 (the focus point in the middle between the focus point 4 and the focus point 6) of the broken lines X and Y, and the shortest distance L6 is the shortest distance of the focus point 6 (the vertex as the other end point of the broken line X) of the broken line X.

[0090] Also, when L1, L2, and L3 > threshold value and L4, L5, and L6 ≤ threshold value, it is determined that within the range of L4 to L6, the two broken lines Y and X (two damages) are "close". Additionally, the number of the focus points P1 to P6 is not limited to the above example.

[0091] Also, the above example illustrates the distance between two linear damages, but the distance between a linear damage and a planar damage can also be obtained in the same way by finding the shortest distance between each focus point of the broken line and the polygon, and making a "close" determination.

[0092] In one aspect of the present invention, a priority order is defined according to the types of damages. When two or more types of damages are detected from the same or close positions, the damages are presented in the priority order. Additionally, the details of the method for presenting damages in the priority order will be described later.

[0093] [Hardware Structure of Maintenance Assistance Device]

[0094] Figure 9 It is a block diagram showing an example of the hardware structure of the maintenance assistance device according to the present invention.

[0095] As Figure 9 shown, for the maintenance assistance device 10, a personal computer or a workstation can be used. The maintenance assistance device 10 in this example mainly includes an image acquisition unit 12, an image database 14, a storage unit 16, an operation unit 18, a CPU (Central Processing Unit), a RAM (Random Access Memory) 22, a ROM (Read Only Memory) 24, and a display control unit 26.

[0096] The image acquisition unit 12 is equivalent to an input / output interface, and in this example, it acquires photographic images of the structure to be maintained, etc. The structure to be maintained includes, for example, structures such as bridges and tunnels.

[0097] The images acquired by the image acquisition unit 12 are, for example, a large number of images (photographic image groups) of the structure taken by a drone (unmanned aerial vehicle) equipped with a camera, a robot, or by hand. Preferably, the photographic image group covers the entire structure, and adjacent photographic images overlap.

[0098] The photographic image group acquired by the image acquisition unit 12 is stored in the image database 14.

[0099] The storage unit 16 is a memory composed of a hard disk device, a flash memory, etc. In the storage unit 16, in addition to the operating system and the maintenance assistance program, information indicating the priority order of damage types, CAD (computer-aided design) data of the structure, and documented damage inspection results, etc. are also stored. Regarding the damage inspection results, they can be stored in different layers according to the types of damage as damage information. In addition, the damage information includes damage diagrams.

[0100] Regarding the CAD data, if there is CAD data of the structure to be inspected, this data can be used. If there is no CAD data of the structure, it can be automatically generated based on the group of photographed images stored in the image database 14.

[0101] When the group of photographed images stored in the image database 14 is taken by a camera mounted on a drone, a three-dimensional point cloud model can be generated. This three-dimensional point cloud model extracts feature points between overlapping photographed images in the group of photographed images, estimates the position and pose of the camera mounted on the drone based on the extracted feature points, and simultaneously estimates the three-dimensional positions of the feature points based on the estimation results of the camera's position and pose.

[0102] There is a Structure from Motion (SfM: three-dimensional reconstruction) method, which tracks the movement of a large number of feature points from the photographed positions of the camera through the group of photographed images during the movement of the drone, and simultaneously estimates the three-dimensional structure (Structure) of the structure and the camera pose (Motion). In recent years, an optimization algorithm called bundle adjustment has been developed, which can perform high-precision output.

[0103] In addition, as the parameters of the camera (focal length, image size of the image sensor, pixel pitch, etc.) required when applying the SfM method, the parameters stored in the storage unit 16 can be used. And the CAD data of the structure can be generated based on the generated three-dimensional point cloud model.

[0104] The operation unit 18 includes a keyboard, a mouse, etc. that are wired or wirelessly connected to the computer. In addition to functioning as an operation unit for giving normal operation instructions to the computer, it also functions as the following operation unit, which edits the damage detection results of the structure detected from the images of the structure through user operations, and sets the priority order of multiple damage types of the structure through user operations. In addition, the details of the editing of the damage detection results and the setting of the priority order of the damage types, etc. will be described later.

[0105] The CPU 20 performs the following processes: a damage detection process, in which various programs stored in the storage unit 16 or the ROM 24 or the like are read out, each unit is centrally controlled, and damage (two or more types of damage) to the structure is detected based on an image of the structure taken; a determination process, in which it is determined whether two or more types of damage are detected from the same or a nearby position; and an output process, in which the damage detection result detected by the damage detection process is output; and so on.

[0106] The damage detection process of detecting two or more types of damage based on an image of the structure taken can be performed by artificial intelligence (AI: artificial intelligence).

[0107] As the AI, for example, a learned model based on a convolutional neural network (CNN: Convolution Neural Network) can be used.

[0108] Figure 10 It is a conceptual diagram showing an embodiment of a damage detection processing unit composed of a CPU or the like.

[0109] In Figure 10 the damage detection processing unit 21 is composed of a plurality of (three in this example) learned models 21A, 21B, and 21C corresponding to a plurality of types of damage.

[0110] Each of the learned models 21A, 21B, and 21C includes an input layer, an intermediate layer, and an output layer, and each layer has a structure in which a plurality of "nodes" are connected by "edges".

[0111] In the input layer of the CNN, an image 13 of the structure taken is input. The intermediate layer has a plurality of sets each including a convolutional layer and a pooling layer, and is a part that extracts features from the image input from the input layer. In the convolutional layer, filtering processing (performing a convolutional operation using a filter) is performed on the nodes located nearby in the previous layer to obtain a "feature map". In the pooling layer, the feature map output from the convolutional layer is reduced to obtain a new feature map. The "convolutional layer" plays a role of extracting features such as edges from the image, and the "pooling layer" plays a role of imparting robustness so that the extracted features are not affected by parallel translation or the like.

[0112] The output layer of the CNN is a part that outputs a feature map representing the features extracted by the intermediate layer. The output layers of the learned models 21A, 21B, and 21C in this example output, for example, the inference results obtained by classifying (segmenting) the regions of each damage of the structure reflected in the image in units of pixels or in units of several pixels as a block as the damage detection results 27A, 27B, and 27C.

[0113] For example, the learned model 21A is a learned model obtained by performing machine learning for detecting damages such as water leakage, surface free lime, and rust juice, and outputs the damage areas of water leakage, surface free lime, and rust juice and the damage type of each damage area as damage detection results (recognition results) 27A. The learned model 21B is a learned model obtained by performing machine learning for detecting damages such as peeling and steel bar exposure, and outputs the damage areas of peeling and steel bar exposure and the damage type of each damage area as damage detection results 27B. The learned model 21C is a learned model obtained by performing machine learning for detecting damages such as cracks and linear free lime, and outputs the damage areas of cracks and linear free lime and the damage type of each damage area as damage detection results 27C.

[0114] In addition, the damage detection processing unit 21 is not limited to the above-described embodiment. For example, it may be configured to have a separate learned model for each damage type, and each learned model outputs the damage area corresponding to each damage type as a damage detection result. In this case, the number of learned models is the same as the number of damage types of the inspection object. Also, it may be configured to have one learned model capable of corresponding to all damage types, and output the damage area and the damage type of each damage area as damage detection results.

[0115] Return to Figure 9 , the CPU 20 outputs the damage detection results detected by the damage detection processing via the display control unit 26 and displays them on the display unit (display) 30, or saves the damage detection results in the storage unit (memory) 16 in the form of a file.

[0116] The RAM 22 is used as the working area of the CPU 20 and as a storage unit for temporarily storing the read program and various data.

[0117] The display control unit 26 is a part that generates display data to be displayed on the display unit 30 and outputs it to the display unit 30. In this example, the damage detection results detected by the CPU 20 are displayed on the display unit 30, and an editing screen for the damage detection results based on the user operation from the operation unit 18 is also displayed on the display unit 30.

[0118] The display unit 30 uses various displays such as a liquid crystal monitor that can be connected to a computer, and displays the damage detection results detected from the image together with the image of the structure taken in from the display control unit 26, and also serves as a part of the user interface together with the operation unit 18.

[0119] The processor of the maintenance assistance device 10 with the above structure, including the CPU 20, performs the above various processes by reading out the maintenance assistance program stored in the storage unit 16 or the ROM 24 and executing the maintenance assistance program.

[0120] <Function of the maintenance assistance device>

[0121] Next, Figure 9 Regarding the function of the maintenance assistance device 10 shown, the structure is exemplified by a bridge.

[0122] Figure 11 It is a perspective view showing an example of the bridge to be inspected.

[0123] As Figure 11 shown, the bridge 1 is composed of various components including the main girder 2 erected between the piers 7, the cross beam 3 provided in the direction orthogonal to the main girder 2 and connecting between the main girders, the sway bracing 4 and the lateral bracing 5 connecting the main girders 2 to each other, and a bridge deck 6 for vehicles and the like to travel is cast on the upper part of the main girder and the like. The bridge deck 6 is usually made of reinforced concrete.

[0124] The bridge deck 6 usually takes the soffit in a rectangular shape divided by the main girder 2 and the cross beam 3 as the basic unit, and in the case of inspecting the damage (cracks, concrete peeling, etc.) of the bridge deck, it is carried out in units of the soffit.

[0125] Each soffit of the bridge deck is one of the components (maintenance units) constituting the structure (bridge). In addition, the maintenance units of the bridge include, in addition to the bridge deck (soffit), the parts / components of the structure (main girder 2, cross beam 3, sway bracing 4, lateral bracing 5, pier 7 (column part / wall part, beam part, corner part / joint part)), etc.

[0126] The CPU 20 of the maintenance assistance device 10, the maintenance assistance program stored in the storage unit 16, the RAM 22 and the ROM 24, the display control unit 26, etc. constitute a processor, and the processor performs various processes shown below.

[0127] The processor performs an image acquisition process, and in this image acquisition process, an image of the maintenance unit is acquired from a plurality of images of the structure (bridge 1) to be inspected stored in the image database 14.

[0128] Figure 12 It is a figure showing an example of an orthoimage corresponding to a soffit which is one of the maintenance units of the bridge.

[0129] The orthographic image is an image obtained by orthographically projecting an image of a structure (caisson ceiling) onto the surface of the caisson ceiling. By extracting a plurality of images corresponding to the caisson ceiling from the group of photographed images stored in the image database 14, performing panoramic synthesis on the extracted plurality of images, and projection-converting the panoramically synthesized image onto the surface of the caisson ceiling, an orthographic image of one caisson ceiling can be generated.

[0130] Figure 10 When the orthographic image (image 13) of the caisson ceiling is input, the damage detection unit 21 shown detects damage to the caisson ceiling based on the input image 13 and outputs the damage detection results 27A to 27C.

[0131] Figure 13 It shows according to Figure 12 The figure is an example of the damage detection result detected from the orthographic image shown.

[0132] Figure 13 The damage detection result shown shows a damage map indicating damage to the caisson ceiling to be inspected.

[0133] In Figure 13 In the damage map shown, five cracks C1 to C5 and the peeling H1 of the concrete are illustrated.

[0134] Figure 13 The damage map shown is represented by a drawing pattern formed by a broken line along each crack C1 to C5 (linear damage) detected on the orthographic image, a drawing pattern formed by a polygon surrounding the peeling H1 (planar damage), or an image filled within the polygon.

[0135] Figure 14 The figure is an example of an orthographic image with the damage map corresponding to the caisson ceiling overlaid.

[0136] Figure 14 The orthographic image with the damage map overlaid shown can be generated by overlaying the Figure 12 damage map shown on the orthographic image Figure 13 shown.

[0137] The damage map can be generated by assigning a color corresponding to the damage type to the damaged part. By overlaying the damage map on the orthographic image, the damaged part can be easily visually recognized.

[0138] Figure 15 The figure is an example of a chart showing the damage quantity table included in the damage detection result.

[0139] Figure 15The shown damage quantity table has items of damage identification information (ID: identification: identification code), damage type, dimensions (width), dimensions (length), and dimensions (area), and for each damage, information corresponding to each item is recorded.

[0140] In the case of cracks as linear damages, the lengths or widths of each crack C1 - C5 are quantified, and in the case of delamination as planar damage, the area of the region of delamination H1 is quantified. These information are recorded in the damage quantity table in association with the damage ID.

[0141] [First Embodiment of Damage Detection Result Output]

[0142] Figure 16 It is a schematic diagram showing an example of the damage detection result of cracks and linear free lime based on the damage detection processing unit and its output processing.

[0143] Figure 16 In (A) of Figure 10 shown, the image 13 is input to the damage detection processing unit 21 (learned model 21C), and the damage detection result indicating the damage regions of the crack B and the linear free lime C2 and the damage type of each damage region is detected by the learned model 21C. In this case, the CPU 20 performs a determination process to determine whether these crack B and linear free lime C2 are detected from the same or adjacent positions respectively.

[0144] In Figure 16 In the example shown in (A) of , for the sake of explanation, the crack B and the linear free lime C2 are arranged and shown, but the linear free lime C2 is in a state where the lime component is blocked in the crack B generated on the concrete component. Therefore, the linear free lime C2 and the crack B have substantially the same shape, and on the linear free lime C2, the crack B is generated at the same position (region) as the linear free lime C2.

[0145] In Figure 16 In the case of the crack B and the linear free lime C2 shown in (A) of , the CPU 20 determines that the crack B and the linear free lime C2 are detected from the same or adjacent positions respectively. And when it is determined that the crack B and the linear free lime C2 are detected from the same or adjacent positions respectively, the CPU 20 performs an output process of outputting the damage detection result according to the priority order of the damage type.

[0146] In this example, as the priority order of the damage type, since the priority order of the linear free lime C2 is set higher than that of the crack B, therefore as shown in Figure 16As shown in (B) thereof, the CPU 20 displays, via the display control unit 26, an image of damage to the area filled with linear free lime C2 on the display unit 30, or outputs, in file form, CAD data of a damage map representing a broken line of the linear free lime C2. The file of the CAD data of the damage map is preferably stored in the storage unit 16 in association with the image in which the damage is detected.

[0147] Figure 17 It is a schematic diagram showing another example of the damage detection result of cracks and linear free lime by the damage detection processing unit and its output processing.

[0148] In Figure 17 In the example shown in (A) thereof, cracks B and linear free lime C2 are detected, and a part of crack B and a part of linear free lime C2 are generated at the same position.

[0149] In this case, the CPU 20 determines that a part of crack B and a part of linear free lime C2 are detected from the same position respectively. And regarding the overlapping part of a part of crack B and linear free lime C2, as Figure 17 shown in (B) thereof, the CPU 20 displays, via the display control unit 26, an image of damage to the area filled with linear free lime C2 on the display unit 30, or outputs, in file form, CAD data of a damage map representing a broken line of the linear free lime C2.

[0150] In addition, regarding the remaining part of crack B that does not overlap with linear free lime C2, the CPU 20 directly displays, via the display control unit 26, an image of damage to the area filled with crack B on the display unit 30, or outputs, in file form, CAD data of a damage map representing a broken line of crack B. And the damage image and CAD data representing crack B and the damage image and CAD data representing linear free lime C2 can preferably be identified by changing the line type (for example, color).

[0151] Figure 18 It is a schematic diagram showing an example of the damage detection result of planar free lime and linear free lime by the damage detection processing unit and its output processing.

[0152] Figure 18 (A) of Figure 10 shows a case where an image 13 is input to the damage detection processing units 21 (learned models 21A, 21C) shown respectively, and the damaged area of planar free lime C1 is detected by the learned model 21A, and the damaged area of linear free lime C2 is detected by the learned model 21C. In this case, the CPU 20 determines whether these planar free lime C1 and linear free lime C2 are detected from the same or adjacent positions respectively.

[0153] In Figure 18 In the example shown in (A) below, a planar free lime C1 and a part of a linear free lime C2 are generated overlapping each other. Therefore, the CPU 20 determines that the planar free lime C1 and the linear free lime C2 are detected from the same or adjacent positions, respectively. And when it is determined that the planar free lime C1 and the linear free lime C2 are detected from the same or adjacent positions, respectively, the CPU 20 outputs a damage detection result in the order of priority of damage types.

[0154] In Figure 18 In the example shown below, since the priority order is set to decrease in the order of the planar free lime C1 and the linear free lime C2, as shown in Figure 18 (B) below, the CPU 20 performs the following: For the overlapping part of the planar free lime C1 and the linear free lime C2, the planar free lime C1 is given priority, and a damage image filling the damaged area of the planar free lime C1 with a specific color is displayed on the display unit 30 via the display control unit 26, and a damage image filling a part of the area of the non-overlapping linear free lime C2 is displayed on the display unit 30 via the display control unit 26. And, as shown in Figure 18 (C) below, for the overlapping part of the planar free lime C1 and the linear free lime C2, the CPU 20 outputs, in file form, the CAD data of the polygon surrounding the damaged area of the planar free lime C1 and the CAD data of a part of the broken line of the non-overlapping linear free lime C2 together.

[0155] Figure 19 is a schematic diagram showing another example of the damage detection result of the planar free lime and the linear free lime based on the damage detection processing unit and its output processing.

[0156] Figure 19 In the example shown in Figure 18 compared with the example shown in

[0157] as shown in Figure 19 (A) below, in the case where a part of the planar free lime C1 and the linear free lime C2 overlap each other, as shown in Figure 19 (B) below, for the overlapping part of the planar free lime C1 and the linear free lime C2, the CPU 20 preferentially displays a damage image filling the area of the linear free lime C2, and similarly, preferentially outputs the CAD data of the broken line of the linear free lime C2 in file form. In this case, neither a damage image nor CAD data is output for the planar free lime C1.

[0158] Figure 20 This is a schematic diagram showing an example of the damage detection results of rust juice, planar free lime, and water leakage based on the damage detection processing unit and its output processing.

[0159] Figure 20 In (A) of Figure 10 As shown, the input image 13 is input to the damage detection processing unit 21 (learned model 21A), and the damage areas of rust juice D, planar free lime C1, and water leakage A are detected by the learned model 21A. In this case, the CPU 20 determines whether these rust juice D, planar free lime C1, and water leakage A are detected from the same or adjacent positions respectively.

[0160] In Figure 20 In the example shown in (A) of

[0161] In Figure 20 In the case of rust juice D, planar free lime C1, and water leakage A shown in (A) of

[0162] In Figure 20 In the example shown in Figure 20 As shown in (B) of Figure 20 Since the priority order is set to decrease in the order of rust juice D, planar free lime C1, and water leakage A, the CPU 20 overlaps the area of planar free lime C1 on the area of water leakage A, and further overlaps the area of rust juice D on the area of planar free lime C1, and displays the damage images with each area filled with a different color according to each damage type on the display unit 30 via the display control unit 26. And as shown in (C) of

[0163] Figure 21 This is a schematic diagram showing another example of the damage detection results of rust juice, planar free lime, and water leakage based on the damage detection processing unit and its output processing.

[0164] Figure 21 The example shown in Figure 20Compared with the example shown, the priority order of rust juice D, planar free lime C1, and water leakage A is set in reverse, and the priority order is set to decrease in the order of water leakage A, planar free lime C1, and rust juice D.

[0165] As Figure 21 shown in (A) of, when all or part of rust juice D, planar free lime C1, and water leakage A overlap each other, as Figure 21 shown in (B) of, the CPU 20 preferentially displays the damage image of the area of water leakage A with the highest filling priority. In this case, the damage images of rust juice D and planar free lime C1 existing inside the area of water leakage A are not displayed. And, as Figure 21 shown in (C) of, the CPU 20 preferentially outputs the CAD data of the polygon surrounding the area of water leakage A with the highest priority in a file. In this case, the CAD data of the polygon surrounding rust juice D and planar free lime C1 existing inside the area of water leakage A is not output.

[0166] In addition, the priority order of damage types is not limited to the above example, and it is preferably set according to the severity of the damage (more severe damage). For example, as damage types, in the case of linear damage including linear free lime and cracks, the priority order of linear free lime is set higher than that of cracks. And, as damage types, in the case of planar damage including steel bar exposure, peeling, rust juice, planar free lime, and water leakage, the priority order is set to decrease in the order of steel bar exposure, peeling, rust juice, planar free lime, and water leakage.

[0167] Moreover, the priority order of damage types can also be appropriately set by the user using the operation unit 18. In this case, the CPU 20 can perform priority order reception processing for receiving the priority order of the damage types of the structure from the operation unit 18 operated by the user, save the received priority order in the storage unit 16, etc., and read out the priority order from the storage unit 16 as needed for use.

[0168] [Second Embodiment of Damage Detection Result Output]

[0169] Figure 22 and Figure 23 are schematic diagrams showing the second embodiment of the damage detection result output GUI (Graphical User Interface).

[0170] Figure 22 is a diagram showing an example of the screen 40 displayed on the display unit 30.

[0171] In this screen 40, a composite image in which a damage image is overlapped on an image of a structure, a check box 42 for selecting a damage type to be displayed, and various icon button classes used in editing and the like are displayed. In addition, the damage detection results for each damage type detected by the damage detection processing unit 21 from the image of the structure can be held as CAD data in a layer structure representing the damage area for each damage type.

[0172] In Figure 22 In the example shown, in the check box 42, all five damage types of water leakage, free lime, rust juice, peeling, and steel bar exposure are selected (five damage types are selected), so the damage images of the above five damage types are overlapped and displayed on the image of the structure displayed on the screen 40.

[0173] Here, the damage image corresponding to the damage type can be generated by filling the damage area with a color corresponding to the damage type according to the CAD data of the layer corresponding to the damage type. The color corresponding to the damage type can use a color set in advance according to the damage type or a color set by the user.

[0174] Figure 23 FIG. is another example of the screen 40 displayed in the display unit 30.

[0175] In Figure 23 In the example shown, compared with Figure 22 the example shown, the damage images displayed on the screen 40 are different.

[0176] In Figure 23 In the screen 40 shown, in the check box 42, three damage types of water leakage, rust juice, and steel bar exposure are selected, so the damage images of the above three damage types are overlapped and displayed on the image of the structure displayed on the screen 40.

[0177] Therefore, in Figure 23 the screen 40 shown, compared with Figure 22 the screen 40 shown, the difference is that the damage images corresponding to free lime and peeling are eliminated.

[0178] According to the second embodiment of the output of the damage detection results, by the user selecting one or more desired damage types, damage images representing the damage of the selected damage types can be displayed. In addition, it is preferable to display only one or more damage types detected from the image in the check box 42. And the display method of the damage area for each damage type is not limited to Figure 22 and Figure 23 the embodiment shown.

[0179] [The Third Embodiment of the Output of the Damage Detection Results]

[0180] Figures 24 to 26 These are schematic diagrams of the GUI of the third embodiment for outputting damage detection results respectively.

[0181] Figure 24 In (A), it is a diagram showing an example of the setting screen 44 for making various settings.

[0182] In Figure 24 In the setting screen 44 shown in (A), the "label" for setting the color of damage etc. is selected. By using this setting screen, the user can set the color of the damage image corresponding to Figure 22 the types of damage shown etc.

[0183] And, in Figure 24 In the setting screen 44 shown in (A), there is a slider 45A used when setting the transparency of the color (filled color) of the damage image and a dialog box 45B for displaying the transparency.

[0184] In Figure 24 In the example shown in (A), the transparency of the filled color of the damage image is set to "10". Additionally, in this example, the transparency when opaque is "0" and the transparency when completely transparent is "100".

[0185] Figure 24 In (B), it is a diagram showing a composite image formed by overlapping a damage image with a transparency of "10" on an image of the structure.

[0186] It is possible to display Figure 24 the composite image shown in (B) by closing the setting screen after setting the transparency to "10" in the setting screen shown in (A) of Figure 24

[0187] Figure 25 In (A), it is a diagram showing a setting screen with the transparency set to "50", Figure 25 In (B), it is a diagram showing a composite image formed by overlapping a damage image with a transparency of "50" on an image of the structure.

[0188] And, Figure 26 In (A), it is a diagram showing a setting screen with the transparency set to "100", Figure 26 In (B), it is a diagram showing a composite image formed by overlapping a damage image with a transparency of "100" on an image of the structure.

[0189] Additionally, in Figure 26 In (B), since the transparency of the filled color is "100", the filled color is completely transparent, but the closed polygons surrounding the damage areas of each damage type are displayed in the colors set according to the damage types.

[0190] By setting the transparency of the color of the filled damage image in this way to display the damage image, the user can visually recognize the image of the structure (damage) covered by the damage image.

[0191] In addition, the second and third embodiments of the damage detection result output can be used in combination.

[0192] [Editing of Damage Detection Results]

[0193] Figure 10 When the damage detection processing unit 21 shown inputs the image 13 of the structure taken, it outputs the damage type and the damage area for each damage type as the damage detection result. However, the damage detection result may sometimes be detected incorrectly or inaccurately.

[0194] For example, the damage area is classified in units of pixels or in units of several pixels grouped together, so there is sometimes a lack of accuracy. Also, cracks detected as two cracks are sometimes preferably joined as one crack. This is because it can be inferred that the cracks are connected inside the concrete.

[0195] Therefore, the CPU 20 performs an edit instruction acceptance process that accepts an edit instruction for the damage detection result by using the operation of the operation unit 18 (e.g., mouse) operated by the user, and performs an edit process for editing the damage detection result according to the accepted edit instruction.

[0196] As an example of editing the damage detection result, in the case of linear damage of the same type and linear damage where the end points of the broken line are close to each other, editing to connect the end points to each other can be considered. Regarding the editing at this time, the distance between the end points of the broken line of the linear damage of the same type can be measured after the damage detection process. If the measured distance is below the threshold value, the end points are automatically connected to each other, or they can be automatically connected according to the user's instruction. The threshold value can use the default value or can be set by the user.

[0197] Also, a threshold value for the length or width of the linear damage or a threshold value for the area of the planar damage can be set to automatically delete the damage detection results smaller than the threshold value. Regarding the deletion of the damage detection result, it can be automatically deleted after the damage detection process or deleted according to the user's instruction. The threshold value can use the default value or can be set by the user.

[0198] Figure 27 and Figure 28These are diagrams showing editing examples of damage detection results. Additionally, when editing damage detection results, as shown in the third embodiment of the damage detection result output, it is preferable to set the transparency of the color filling the damaged image to a relatively high level so that the image of the structure is easily visually recognizable.

[0199] Figure 27 This is a diagram showing a method of adding vertices to a polygon enclosing a damaged area.

[0200] The polygon is drawn by connecting multiple vertices along the damaged area (in Figure 27 it is represented by a quadrilateral).

[0201] When adding vertices to this polygon, as shown in (A) of Figure 27 , align the cursor of the mouse with the line of the polygon where you want to add a vertex, right-click the mouse, and select [Add] from the context menu. Thus, as shown in (B) of Figure 27 , a new vertex can be added to the line of the polygon.

[0202] Furthermore, by dragging the added vertex and moving it to the edge of the original damaged area, the polygon enclosing the damaged area can be edited.

[0203] Figure 28 This is a diagram showing a method of deleting vertices from a polygon enclosing a damaged area.

[0204] When deleting vertices from this polygon, as shown in (A) of Figure 28 , align the cursor of the mouse with the vertex you want to delete, right-click the mouse (to make the vertex in a selected state), and select [Delete] from the context menu. Thus, as shown in (B) of Figure 28 , vertices can be deleted from the polygon.

[0205] As shown in (B) of Figure 28 , when deleting a vertex from the polygon, connect the lines of the polygon between the vertices before and after the deleted vertex, thereby editing the polygon enclosing the damaged area.

[0206] The above editing examples illustrate the addition, deletion, etc. of vertices in the polygon of a planar damage, but the addition, deletion, etc. of vertices in the broken line of a linear damage can also be performed in the same manner.

[0207] Furthermore, as an editing function, it has functions such as making the entire broken line or polygon in a selected state by clicking on the line connecting the vertices, a function of deleting the entire broken line or polygon at once, or a function of manually newly adding a broken line or polygon to the missed detection part of the damage.

[0208] [Maintenance Assistance Method]

[0209] Figure 29 It is a flowchart showing an embodiment of the maintenance assistance method involved in the present invention.

[0210] Figure 29 The processing of each step shown is performed, for example, by Figure 9 a processor composed of a CPU 20, etc. of the maintenance assistance device 10 shown.

[0211] In Figure 29 , the processor acquires an image of the structure to be maintained taken by the image acquisition unit 12 or the image database 14, etc. (step S10).

[0212] The damage detection processing unit 21 ( Figure 10 ) detects damage to the structure based on the image acquired in step S10 (step S12).

[0213] The processor determines whether damage is detected through the damage detection performed in step S12 (step S14). When damage is detected ("yes" case), it determines whether two or more types of damage are detected (step S16).

[0214] When it is determined in step S16 that two or more types of damage are detected ("yes" case), the processor further determines whether two or more types of damage are detected from the same or adjacent positions among the two or more types of damage (step S18).

[0215] And when it is determined in step S18 that two or more types of damage are detected from the same or adjacent positions ("yes" case), the processor outputs the damage detection result according to the priority order of the damage types (step S20). The output of the damage detection result is performed, for example, by overlapping the damage image on the image, or separately displaying the damage image on the display unit, or outputting the CAD data representing the damage diagram in the form of a file.

[0216] On the other hand, when two or more types of damage are not detected in step S16 ("no" case), that is, when only one type of damage is detected, or when it is determined in step S18 that two or more types of damage are not detected from the same or adjacent positions ("no" case), it transfers to step S22. In step S22, the damage detection result of one type or two or more types is directly output.

[0217] [Other]

[0218] The hardware of the maintenance assistance device related to the present invention can be constituted by various processors. The various processors include a general-purpose processor that executes a program and functions as various processing units, namely a CPU (Central Processing Unit), a processor whose circuit structure can be changed after manufacturing, such as an FPGA (Field Programmable Gate Array), that is, a programmable logic device (Programmable Logic Device; PLD), and a processor with a circuit structure specifically designed to execute specific processing, such as an ASIC (Application Specific Integrated Circuit), that is, a dedicated circuit, etc. One processing unit constituting the maintenance assistance device can be constituted by one of the above various processors, or can be constituted by two or more processors of the same kind or different kinds. For example, one processing unit can also be constituted by a combination of multiple FPGAs or a combination of a CPU and an FPGA. Also, multiple processing units can be constituted by one processor. As an example of constituting multiple processing units by one processor, first, there is the following method: as represented by a computer such as a client or a server, a processor is constituted by a combination of one or more CPUs and software, and this processor functions as multiple processing units. Second, there is the following method: as represented by a system on chip (System On Chip; SoC), etc., a processor that uses one IC (Integrated Circuit) chip to implement the functions of the entire system including multiple processing units is used. Thus, as a hardware structure, various processing units are constituted by using one or more of the above various processors. Moreover, more specifically, the hardware structure of these various processors is a circuitry formed by combining circuit elements such as semiconductor elements.

[0219] Moreover, the present invention includes a maintenance assistance program and a storage medium storing the maintenance assistance program. By being installed in a computer, the maintenance assistance program causes the computer to function as the maintenance assistance device related to the present invention.

[0220] In addition, the present invention is not limited to the above-described embodiments, and various modifications can of course be made without departing from the spirit of the present invention.

Claims

1. An inspection assistance device, which includes a processor, The processor performs the following processes: Damage detection process, detecting damage to the structure by using an image obtained by photographing the structure to be inspected; Determination process, determining whether two or more types of damage are detected from the same or adjacent positions; and Output process, when it is determined by the determination process that two or more types of damage are detected from the same or adjacent positions, outputting the damage detection results in the order of priority of the damage types.

2. The inspection assistance device according to claim 1, wherein, In the damage detection process, the damage area and the damage type of each damage area are detected based on the image, The determination process determines whether two or more types of damage are detected in the same or adjacent damage areas, In the output process, when it is determined by the determination process that two or more types of damage are detected in the same or adjacent damage areas, outputting the damage detection result of the damage type with the highest priority so that the user can visually recognize it as the damage detection result of the same or adjacent damage areas.

3. The inspection assistance device according to claim 1 or 2, wherein, The adjacent position is a position where the distance between the two or more types of damage is equal to or less than a threshold value.

4. The inspection assistance device according to claim 1 or 2, wherein, In the damage detection process, when the image is input, a learned model that outputs the damage area and the damage type of each damage area as recognition results is executed.

5. The inspection assistance device according to claim 1 or 2, wherein, In the output process, different drawing patterns are output in the case where the damage type is a linear damage and in the case where the damage type is a planar damage.

6. The inspection assistance device according to claim 5, wherein, In the output process, in the case where the damage type is a linear damage, a damage diagram representing a line that does not close the linear damage is output, and in the case where the damage type is a planar damage, a damage diagram representing a closed line that surrounds the planar damage is output.

7. The inspection assistance device according to claim 5, wherein, In the output process, in the case where the damage type is a linear damage, a damage image that at least fills the linear damage is output, and in the case where the damage type is a planar damage, a damage image that at least fills the planar damage is output.

8. The inspection assistance device according to claim 1 or 2, wherein, The output process outputs and displays the damage detection results on a display, or stores the damage detection results in a memory in the form of a file.

9. The inspection assistance device according to claim 1 or 2, wherein, The priority order of the damage types among the two or more types of damage is a priority order preset according to the severity of the damage indicated by each of the two or more types of damage.

10. The inspection assistance device according to claim 9, wherein, As the damage type, in the case of a linear damage including linear free lime and cracks, the priority order of the linear free lime is higher than that of the cracks.

11. The maintenance assistance device according to claim 9, wherein, As the types of damage, in the case of including steel bar exposure, peeling, rust juice, planar free lime, and planar damage such as water leakage, the priority order is set to decrease in the order of steel bar exposure, peeling, rust juice, planar free lime, and water leakage.

12. The maintenance assistance device according to claim 1 or 2, wherein, The processor performs a priority acceptance process of accepting the priority order of the types of damage to the structure from the operation unit operated by the user, The priority order of the types of damage is the priority order accepted from the user via the operation unit.

13. The maintenance assistance device according to claim 1 or 2, wherein, The processor performs the following processes: An editing instruction acceptance process of accepting an editing instruction for the damage detection result from the operation unit operated by the user; and An editing process of editing the damage detection result according to the accepted editing instruction.

14. The maintenance assistance device according to claim 1 or 2, wherein, The damage detection result includes a damage quantity table, and the damage quantity table has items of damage identification information, types of damage, and dimensions, and information corresponding to each item is recorded for each detected damage.

15. A maintenance assistance method, in which a processor performs maintenance assistance for a structure to be maintained, Each process of the processor includes the following steps: Detect damage to the structure based on an image obtained by photographing the structure; Determine whether two or more types of damage are detected from the same or adjacent positions; and Output the detected damage detection result, and when it is determined in the determination step that two or more types of damage are detected from the same or adjacent positions, output the damage detection result according to the priority order of the types of damage.

16. A recording medium records a maintenance assistance program, and the maintenance assistance program causes a computer to execute a method for performing maintenance assistance for a structure to be maintained, and the method includes the following steps: Detect damage to the structure based on an image obtained by photographing the structure; Determine whether two or more types of damage are detected from the same or adjacent positions; and Output the detected damage detection result, and when it is determined in the determination step that two or more types of damage are detected from the same or adjacent positions, output the damage detection result according to the priority order of the types of damage.

Images