Inspection systems, methods for inspecting structures, computer programs, and storage media

Abstract

To provide an inspection system capable of efficiently inspecting an inspection object portion of a structure, a structure inspection method, a computer program, and a storage medium.SOLUTION: The inspection system comprises: an inspection unit 56 which inspects an inspection object portion of a crane 100 based on a plurality of images captured by an imaging unit 15; and a display control unit 59 which performs processing for displaying on a display unit 61 a result of the inspection of the inspection object portion and the images used for the inspection of the inspection object portion.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an inspection system, a method for inspecting a structure, a computer program, and a storage medium. [Background Art]

[0002] According to the disclosure of Patent Document 1, an imaging device mounted on an unmanned aerial vehicle images a large-scale structure, and a change detection device uses a difference between image data generated by the imaging device to detect a change in the state of the structure, for example, the occurrence of rust. [Prior Art Literature] [Patent Literature]

[0003] [Patent Document 1] Japanese Patent Laying-Open No. 2018-185208 [Summary of the Invention] [Problem to be Solved by the Invention]

[0004] With the technology of Patent Document 1, inspection of a specific portion of a structure (for example, detection of a state change of the portion or detection of an abnormality) cannot be performed efficiently.

[0005] An object of the present invention is to provide an inspection system, a method for inspecting a structure, a computer program, and a storage medium that can more efficiently inspect an inspection target portion of a structure. [Means for Solving the Problem]

[0006] (1) The present invention One aspectThe system for inspecting a structure comprises: an inspection unit that inspects a target part of the structure based on a plurality of images captured by an imaging unit; an extraction unit that extracts images of a plurality of target parts captured in the plurality of images for each target part; and a display control unit that processes the results of the inspection of the target part and the plurality of images used for the inspection of the target part to be displayed on a display unit, wherein the inspection unit performs an inspection of a type corresponding to the type of target part for each of the same target parts based on the images of the target part extracted by the extraction unit. (2) This invention Another aspect of this is, A structural inspection system comprising: an inspection unit that inspects a target part of the structural based on a plurality of images captured by an imaging unit; and a display control unit that processes the results of the inspection of the target part and the plurality of images used for the inspection of the target part to be displayed on a display unit, wherein the inspection unit performs a plurality of preliminary inspections based on the plurality of images, determines a final inspection result based on the results of the plurality of preliminary inspections, and the display control unit causes the final inspection result to be displayed on the display unit. (3) Another aspect of the present invention is a method for inspecting a structure, comprising: an inspection step of inspecting a target portion of the structure based on a plurality of images captured by an imaging unit; an extraction step of extracting images of a plurality of target portions captured in the plurality of images for each target portion; and a display control step of performing processing to display the inspection results of the target portion and the plurality of images used for inspecting the target portion on a display unit, wherein in the inspection step, an inspection of a type corresponding to the type of target portion is performed for each of the same target portions based on the images of the target portion extracted by the extraction step. (4) Another aspect of the present invention is a method for inspecting a structure, comprising: an inspection step of inspecting a target portion of the structure based on a plurality of images captured by an imaging unit; and a display control step of performing processing to display the result of the inspection of the target portion and the plurality of images used to inspect the target portion on a display unit, wherein in the inspection step, a plurality of preliminary inspections are performed based on the plurality of images, a final inspection result is determined based on the results of the plurality of preliminary inspections, and in the display control step, the final inspection result is displayed on the display unit. (5) Another aspect of the present invention is a computer program that causes a computer to perform the following steps: an inspection step of inspecting a target part of a structure based on a plurality of images captured by an imaging unit; an extraction step of extracting images of a plurality of target parts captured in the plurality of images for each target part; and a display control step of performing processing to display the inspection results of the target parts and the plurality of images used for inspecting the target parts on a display unit, wherein in the inspection step, the computer performs an inspection of a type corresponding to the type of target part for each of the same target parts based on the images of the target parts extracted by the extraction step. (6) Another aspect of the present invention is a computer program that causes a computer to perform an inspection step of inspecting a target part of a structure based on a plurality of images captured by an imaging unit, and a display control step of performing processing to display the result of the inspection of the target part and the plurality of images used to inspect the target part on a display unit, wherein the computer performs a plurality of preliminary inspections based on the plurality of images in the inspection step, determines a final inspection result based on the results of the plurality of preliminary inspections, and the computer displays the final inspection result on the display unit in the display control step. [Effects of the Invention]

[0007] According to the present invention, inspection of the target part of a structure can be performed more efficiently. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram of the inspection system according to the first embodiment. [Figure 2] This is a perspective view showing a crane. [Figure 3] It is a diagram showing an example of a selected image used for inspection. [Figure 4] It is a diagram showing an example of a data table in which types of inspection target sites and types of inspections are associated with each other. [Figure 5] It is a diagram showing an example of a screen displayed on a display unit as a result of inspection based on the selected images shown in Figs. 3(a) to 3(c). [Figure 6] It is a diagram showing an example of a screen displayed on a display unit as a result of inspection based on the selected images shown in Figs. 3(a) to 3(c). [Figure 7] It is a diagram showing an example of a screen displayed on a display unit as a result of inspection based on the selected image shown in Fig. 3(a). [Figure 8] It is a diagram showing an example of a selected image used for inspection.

Mode for Carrying Out the Invention

[0009] Hereinafter, each embodiment will be described in detail with reference to the drawings.

[0010] <<First Embodiment>> <1. Overview of Crane and Inspection System> Fig. 1 is a block diagram showing the inspection system according to the first embodiment. Fig. 2 is a perspective view showing a crane 100 which is a structure.

[0011] As shown in Fig. 2, the crane 100 includes a lower traveling body 101, an upper rotating body 102, a boom 103, a main hoisting winch 104, a guide sheave unit 105, a point sheave 106, a main hoisting rope 107, a main hoisting hook 108, an auxiliary hoisting winch 109, a point sheave 110, an auxiliary hoisting rope 111, an auxiliary hoisting hook 112, a luffing winch 113, a luffing rope 114, and the like.

[0012] The lower running body 101 is a self-propelled crawler. The upper slewing body 102 is rotatably mounted on the lower running body 101. The boom 103 is luffably mounted on the upper slewing body 102. The main hoisting winch 104, auxiliary hoisting winch 109, and luffing winch 113 are mounted on the upper slewing body 102 behind the boom 103. The guide sheave unit 105 is rotatably mounted to the rear of the upper end of the boom 103. The point sheaves 106 and 110 are rotatably mounted to the front of the upper end of the boom 103 via brackets. The main hoisting rope 107 is unfurled from the main hoisting winch 104 and wound around the guide sheave unit 105 and point sheave 106 in order, and hangs down from the point sheave 106. The main hoisting hook 108 is provided at the lower end of the hanging main hoisting rope 107. The main hoisting winch 104 raises and lowers the main hoisting hook 108 and the load attached to it by winding in and unwinding the main hoisting rope 107. The auxiliary hoisting rope 111 is unwinded from the auxiliary hoisting winch 109 and wound around the guide sheave unit 105 and the point sheave 110 in order, and hangs down from the point sheave 110. The auxiliary hoisting hook 112 is provided at the lower end of the hanging auxiliary hoisting rope 111. The auxiliary hoisting winch 109 raises and lowers the auxiliary hoisting hook 112 and the load attached to it by winding in and unwinding the auxiliary hoisting rope 111. The luffing rope 114 consists of a winding rope and a pendant rope provided from the luffing winch 113 to the upper end of the boom 103. The luffing winch 113 luffs the boom 103 by winding in and unwinding the luffing rope 114.

[0013] The inspection system shown in Figure 1 is a system that images part or all of the crane 100 from various shooting positions and angles, inspects the crane 100 based on multiple captured images, and displays the inspection results. Below, an example is described in which the inspection target part of the crane 100 is the guide sheave unit 105. Each captured image shows not only the guide sheave unit 105, but also parts of the crane 100 surrounding the guide sheave unit 105. For example, parts of the boom 103 and the main hoisting rope 107 are captured together with the guide sheave unit 105. These boom 103 and main hoisting rope 107 are also inspection target parts. In other words, each captured image shows multiple inspection target parts. Note that the inspection target parts are not limited to the guide sheave unit 105, boom 103 and main hoisting rope 107, but may also be various parts such as the jib, wire, cab, slewing frame, traveling body, various connecting parts, and various electrical components.

[0014] The inspection system comprises an aircraft 10 and an inspection device 50. The aircraft 10 and the inspection device 50 will be described in detail below.

[0015] <2. Flying Object> The aircraft 10 is a multirotor, commonly known as a drone. The aircraft 10 flies around the crane 100, particularly around the guide sheave unit 105 which is the part to be inspected, and images the guide sheave unit 105, the boom 103, and the main hoisting rope 107, and transmits the images obtained from the imaging to the inspection device 50. Flight of the aircraft 10 refers to movement in all directions (up, down, forward, backward, left, and right), nose rotation, and hovering. Nose rotation refers to yawing. The direction in which the nose of the aircraft 10 is pointing is defined as forward, and hereafter, the direction in which the nose of the aircraft 10 is pointing will be referred to as the orientation of the aircraft 10.

[0016] The aircraft 10 comprises multiple propellers 11, multiple drive units 12, a measurement unit 13, a movement control unit 14, an imaging unit 15, a storage unit 16, a communication unit 17, and a processing unit 18.

[0017] The measurement unit 13 measures the attitude of the aircraft 10, i.e., the yaw angle, roll angle, and pitch angle, using a gyro sensor or angular velocity sensor, and also measures the speed of the aircraft 10 using a gyro sensor or angular velocity sensor. The measurement unit 13 transmits the measured attitude and speed information to the movement control unit 14. The movement control unit 14 controls the drive unit 12 based on the attitude and speed information input from the measurement unit 13. The drive unit 12 drives the propellers 11 individually under the control of the movement control unit 14. The aircraft 10 flies through the air as the propellers 11 are driven.

[0018] The measurement unit 13 measures the position of the aircraft 10 using a GNSS (Global Navigation Satellite System), altimeter, or beacon, or a combination of two or more of these, and measures the orientation of the aircraft 10 using a magnetic sensor, gyro sensor, or GNSS, or a combination of two or more of these. The measurement unit 13 transmits the information of the measured position and measured orientation to the processing unit 18.

[0019] The processing unit 18 comprehensively controls the measurement unit 13, the movement control unit 14, the imaging unit 15, the storage unit 16, and the communication unit 17.

[0020] The processing unit 18 is a microcomputer consisting of a CPU, RAM, and a storage medium. The processing unit 18 has a flight path program in its storage medium, which is arranged in chronological order for multiple shooting positions and the shooting angle at each shooting position. The processing unit 18 monitors the measurement position and measurement direction by the measurement unit 13 and outputs commands to the movement control unit 14 according to the flight path program. The movement control unit 14 controls the drive unit 12 according to the commands from the processing unit 18, so that the aircraft 10 moves sequentially to each shooting position and turns its nose to face the shooting angle at each shooting position. After the aircraft 10 turns its nose at each shooting position, the processing unit 18 outputs an imaging command to the imaging unit 15, so that the imaging unit 15 images multiple inspection target parts of the crane 100. By the processing unit 18 controlling the movement control unit 14 and the imaging unit 15 as described above, the aircraft 10 flies in such a way that at least one of the shooting position and shooting angle by the imaging unit 15 changes.

[0021] The imaging unit 15 is installed on the aircraft 10. The imaging unit 15 may be capable of tilting, panning, or both, or its orientation may be fixed. The imaging unit 15 is, for example, a monocular camera and comprises an optical system, an image sensor, and a signal processing unit. The optical system forms an image of an object to be photographed in front of the aircraft 10, for example, the inspection target part of the crane 100, onto the image sensor. The image sensor captures the image formed by the optical system by photoelectric conversion. The signal processing unit processes the image captured by the image sensor at the timing of the imaging command from the processing unit 18 to generate an image. Note that the wavelength of the image to be formed is not limited to the visible light region, but may also be an image in the invisible light region, such as the ultraviolet (UV) or infrared (IR) region.

[0022] The processing unit 18 records the image captured by the imaging unit 15, that is, the image generated by the signal processing unit of the imaging unit 15, in the storage unit 16, and also records information on the shooting position and shooting angle in association with that image in the storage unit 16. Here, the information on the shooting position and shooting angle corresponds to the information on the measurement position and measurement direction measured by the measurement unit 13 when the associated image was captured by the imaging unit 15, or corresponds to the shooting position and shooting position of the flight path program.

[0023] The communication unit 17 transmits the image recorded in the storage unit 16, along with the corresponding information on the shooting position and shooting angle, to the external computer 51.

[0024] <3. Inspection Equipment> The inspection device 50 is installed, for example, in the operator's cab of the crane 100 or in an office near the crane 100. The inspection device 50 may be a fixed type or a portable type.

[0025] The inspection device 50 inspects multiple inspection targets of the crane 100, one for each same inspection target, based on multiple images transmitted by the communication unit 17 of the aircraft 10. The inspection device 50 then displays the inspection results for each same inspection target, the type of inspection for each same inspection target, and the multiple images used for the inspection.

[0026] The inspection device 50 includes a computer 51, a display unit 61, and an input unit 62.

[0027] The display unit 61 is connected to the computer 51. The display unit 61 is, for example, a liquid crystal display or an EL display, and displays images according to the video signals transferred from the computer 51.

[0028] The input unit 62 is connected to the computer 51. The input unit 62 is, for example, a keyboard, mouse, touch panel, or push buttons, or a combination of two or more of these. When the input unit 62 is operated by the user, it transmits signals corresponding to the operation to the computer 51. In this way, the user can input various types of information to the computer 51 via the input unit 62, and the computer 51 can acquire the information input by the user.

[0029] Computer 51 comprises hardware such as a CPU, RAM, GPU, storage medium 63, system bus, and communication unit. The communication unit of computer 51 communicates with the communication unit 17 of the aircraft 10. The storage medium 63 of computer 51 stores a computer program 64 that can be executed by the CPU. By executing the computer program 64, computer 51 functions as an image storage unit 52, an image selection unit 53, an extraction / recognition unit 54, an inspection type selection unit 55, an inspection unit 56, and a display control unit 59. The image storage unit 52, image selection unit 53, extraction / recognition unit 54, inspection type selection unit 55, inspection unit 56, and display control unit 59 are software modules realized by the CPU executing the computer program 64. The inspection unit 56 has a preliminary inspection unit 57 and an aggregation unit 58. The preliminary inspection unit 57 and the aggregation unit 58 are software modules realized by the CPU executing subprograms included in the computer program 64 as part of the computer program 64.

[0030] The image storage unit 52 records the received images and the corresponding shooting position and shooting angle information in the storage medium 63 each time it receives an image and the corresponding shooting position and shooting angle information transmitted by the communication unit 17 of the aircraft 10. Alternatively, after the aircraft 10 has finished flying, the multiple images recorded in the storage unit 16 and the corresponding shooting position and shooting angle information for each image may be sent together from the communication unit 17 of the aircraft 10 to the communication unit of the computer, and the image storage unit 52 may store these images and the corresponding shooting position and shooting angle information in the storage medium 63. Or, after the aircraft 10 has finished flying, the user may use a portable storage medium or wired communication to copy the multiple images recorded in the storage unit 16 and the corresponding shooting position and shooting angle information for each image together to the storage medium 63 of the computer 51, in which case the copying function of the computer 51 corresponds to the image storage unit 52.

[0031] The image selection unit 53 selects and reads two or more images for inspection from among multiple images recorded on the storage medium 63. Specifically, two or more pre-set selection positions and selection angles are stored in a database on the storage medium 63 of the computer 51, and the image selection unit 53 selects and reads an image that corresponds to the shooting position and shooting angle corresponding to the selection position and selection angle recorded on the storage medium 63. Alternatively, the image selection unit 53 estimates the shooting position and shooting angle of an image by image recognition processing of the image recorded on the storage medium 63, and then selects and reads an image that is estimated to correspond to the shooting position and shooting angle corresponding to the selection position and selection angle stored in the database on the storage medium 63. Alternatively, when the user selects an image using the input unit 62, the image selection unit 53 selects and reads the image according to the operation signal from the input unit 62. Hereinafter, the image selected by the image selection unit 53 will be referred to as the selected image.

[0032] Figures 3(a) to 3(c) show examples of selected images. As shown in Figure 3(a), the selected image includes the inspection target area: the guide sheave unit 105, the boom 103, and the main winding rope 107. The selected image includes images 211 of the guide sheave unit 105, 212 of the boom 103, and 213 of the main winding rope 107. The selected image shown in Figure 3(b) was acquired from a more distal position than the selected image shown in Figure 3(a). The selected image shown in Figure 3(b) includes images 221 of the guide sheave unit 105, 222 of the boom 103, and 223 of the main winding rope 107. The selected image shown in Figure 3(c) was acquired from a more proximal position than the selected image shown in Figure 3(a). The selected image shown in Figure 3(c) includes images 231 of the guide sheave unit 105 and 232 of the boom 103.

[0033] The extraction and recognition unit 54 uses a machine learning-trained model to perform image recognition processing, extracting images of the target areas for inspection from each selected image selected by the image selection unit 53, and recognizing the type of target area for each image of the target area. To give a specific example, in the case of the selected images shown in Figure 3(a), the extraction and recognition unit 54 extracts images 211, 212, and 213, and recognizes the types of target areas in images 211, 212, and 213 as "sheave," "boom," and "rope," respectively. In the case of the selected images shown in Figure 3(b), the extraction and recognition unit 54 extracts images 221, 222, and 223, and recognizes the types of target areas in images 221, 222, and 223 as "sheave," "boom," and "rope," respectively. In the case of the selected image shown in Figure 3(c), the extraction and recognition unit 54 extracts images 231 and 232 and recognizes the types of inspection target parts in images 231 and 232 as type "sheave" and "boom," respectively. Note that the types of inspection target parts are not limited to types "sheave," "boom," and "rope," but various types can be used. For example, types "jib," "wire," "cab," "slewing frame," "vehicle," "house," and "scaffolding" can be used as types of inspection target parts.

[0034] The extraction and recognition unit 54 associates images of the same target area extracted from each selected image with each other. To give a specific example, if images 211, 221, and 231 in the selected images shown in Figures 3(a) to (c) are images of the same target area, the extraction and recognition unit 54 associates these images 211, 221, and 231 with each other. Similarly, the extraction and recognition unit 54 associates images 212, 222, and 232 in the selected images shown in Figures 3(a) to (c) with each other, and associates images 213 and 223 in the selected images shown in Figures 3(a) to (c) with each other.

[0035] The examination type selection unit 55 selects the type of examination for each image of the area to be examined in each selected image, according to the type of area to be examined recognized by the extraction and recognition unit 54. Specifically, a data table is stored in the storage medium 63, which associates the types of areas to be examined with the types of examinations. The examination type selection unit 55 refers to this data table and selects the corresponding type of examination from the types of areas to be examined recognized by the extraction and recognition unit 54.

[0036] To give specific examples of inspection types, from an algorithmic perspective, inspections can be broadly categorized into "object detection," "semantic segmentation," and "autoencoder." However, the broad categories of inspections are not limited to "object detection," "semantic segmentation," and "autoencoder," and various other types can be employed.

[0037] Object detection is an algorithm using a trained model that detects whether or not an object with specific attributes exists within an image, and also infers the position and extent of each detected object. Examples of inspections that employ object detection algorithms include determining whether a target part or its components are missing, whether or not there are scratches on the target part, and whether or not the target part is properly fixed. Object detection can be further subdivided, and the types of sub-sub-sub-sub-sub-subjects of object detection include algorithms using R-CNN (Region Based Convolutional Neural Networks) as a trained model, algorithms using neural networks that employ YOLO (You Only Look Once) technology as a trained model, and neural network algorithms that employ SSD (Single Shot MultiBox Detector) technology.

[0038] Semantic segmentation is a trained model algorithm that classifies what is depicted in pixels or regions within an image, either pixel-wise or region-wise. Inspections employing semantic segmentation algorithms are suitable for determining whether the condition of a target area has changed or whether defects such as rust, corrosion, or scratches have occurred in the target area. Semantic segmentation can be further subdivided into three types: algorithms using a trained neural network model employing U-Net technology, algorithms using a trained neural network model employing SegNet technology, and neural network algorithms employing PSPNet technology.

[0039] An autoencoder is a neural network algorithm that compresses the dimensions of an image using an encoder and then restores it to its original dimensions using a decoder. Autoencoder algorithms are suitable for determining whether the state of a target area has changed, whether the target area has changed, and whether the target area has deformed. Autoencoders are further classified into several types, including algorithms that use a pre-trained neural network model employing stacked autoencoder technology, neural network algorithms that employ variational autoencoder technology, and algorithms that use a pre-trained Generative Adversarial Network (GAN) model.

[0040] As described above, the inspection type selection unit 55 refers to a data table, in which, as shown in Figure 4, the inspection target part type "sheave" is associated with the inspection type "object detection", the inspection target part type "boom" is associated with the inspection type "semantic region segmentation", and the inspection target part type "rope" is associated with the inspection type "autoencoder". Therefore, for the selected image shown in Figure 3(a), the inspection type selection unit 55 selects the inspection types "object detection", "semantic region segmentation", and "autoencoder" for images 211, 212, and 213, respectively. For the selected image shown in Figure 3(b), the inspection type selection unit 55 selects the inspection types "object detection", "semantic region segmentation", and "autoencoder" for images 221, 222, and 223, respectively. For the selected image shown in Figure 3(c), the inspection type selection unit 55 selects the inspection types "object detection" and "semantic region segmentation" for images 231 and 232, respectively. It should be noted that the correspondence between the type of body part to be examined and the type of examination in the data table is not limited to the example shown in Figure 4, and various other correspondences can be adopted in the data table.

[0041] The preliminary inspection unit 57 performs the type of inspection selected by the inspection type selection unit 55 on each selected image of the target area, based on the image of the target area extracted by the extraction and recognition unit 54. Note that the inspection performed by the preliminary inspection unit 57 is a preliminary inspection.

[0042] The inspection process of the preliminary inspection unit 57 will be explained below with specific examples. In the case of the selected image shown in Figure 3(a), the preliminary inspection unit 57 uses a trained model employing the selected type "object detection" algorithm for image 211, inputs image 211 into the trained model, and outputs the inspection result from the trained model. The preliminary inspection unit 57 uses a trained model employing the selected type "semantic domain segmentation" algorithm for image 212, inputs image 212 into the trained model, and outputs the inspection result from the trained model. The preliminary inspection unit 57 uses a trained model employing the selected type "autoencoder" algorithm for image 213, inputs image 213 into the trained model, and outputs the inspection result from the trained model. The same applies to the selected images shown in Figures 3(b) and (c). In other words, in the case of the selected image shown in Figure 3(b), the preliminary inspection unit 57 uses pre-trained models of type "object detection," "semantic segmentation," and "autoencoder" for inspection based on images 221, 222, and 223, respectively. In the case of the selected image shown in Figure 3(c), the preliminary inspection unit 57 uses pre-trained models of type "object detection" and "semantic segmentation" for inspection based on images 231 and 232, respectively. All of the pre-trained models described above are selected from a library that is optimally designed to match the type, location, range, or color of the target area to be inspected, based on the type or feature quantities of the inspection subcategory, or both.

[0043] The test results, based on images of each body part being examined, may be binary (abnormal or normal), multi-valued (a numerical representation of normality or abnormality), or, in the case of classification problems, a sequence of numbers indicating the likelihood of a certain state.

[0044] The aggregation unit 58 aggregates the inspection results from the preliminary inspection unit 57 based on the images of the target area in each selected image for each target area, and thereby determines the final inspection result based on the results of multiple inspections by the preliminary inspection unit 57. The aggregation result from the aggregation unit 58 is the inspection result from the inspection unit 56, and is also the final inspection result.

[0045] Using the selected images shown in Figures 3(a) to 3(c), the aggregation by the aggregation unit 58 will be explained in detail. When images 211, 221, and 231 are images of the same inspection target part, the "guide sheave unit," the aggregation unit 58 aggregates the inspection results based on images 211, 221, and 231. When images 212, 222, and 232 are images of the same inspection target part, the "boom," the aggregation unit 58 aggregates the inspection results based on images 212, 222, and 232. When images 231 and 232 are images of the same inspection target part, the "main winding rope," the aggregation unit 58 aggregates the inspection results based on images 231 and 232.

[0046] Specific examples of aggregation performed by the aggregation unit 58 are as follows (1) or (2).

[0047] (1) If the inspection result by the preliminary inspection unit 57 is binary, the aggregation unit 58 takes a majority vote of the inspection results of the images of the same inspection target area by the preliminary inspection unit 57 and determines whether it is normal or abnormal based on that majority vote. For example, in the case of the selected images shown in Figures 3(a) to (c), images 211, 221, and 231 are images of the same inspection target area "guide sieve unit", so the aggregation unit 58 takes a majority vote of the inspection results of images 211, 221, and 231 by the preliminary inspection unit 57 and determines whether it is normal or abnormal based on that majority vote. If the number of inspection results is even, and the result of the majority vote is equal, the determination of whether it is normal or abnormal is arbitrary, and the aggregation unit 58 determines it to be either abnormal or normal. Furthermore, the aggregation unit 58 counts the number of selected images for each same target area that show abnormal or normal results from the preliminary inspection unit 57. For example, in the case of the selected images shown in Figures 3(a) to 3(c), the aggregation unit 58 counts "0" as the number of selected images for which the preliminary inspection unit 57 showed abnormal results for images 211, 221, and 231. Furthermore, the aggregation unit 58 calculates, for each same target area, the ratio of the number of selected images selected by the image selection unit 53 to the number of selected images in which the inspection result of the image of the target area by the preliminary inspection unit 57 is either abnormal or normal. For example, in the case of the selected images shown in Figures 3(a) to 3(c), the aggregation unit 58 calculates the ratio of the number of selected images in which the inspection result of images 211, 221, and 231 is abnormal to the number of selected images in which there are 3.

[0048] (2) If the inspection results from the preliminary inspection unit 57 are multi-value, the aggregation unit 58 calculates the total or average value of the inspection results of the images of the inspection target area by the preliminary inspection unit 57 for each same inspection target area, and determines whether it is normal or abnormal depending on whether the average value or total value exceeds a predetermined threshold. In the case of the selected images shown in Figures 3(a) to (c), images 211, 221, and 231 are images of the same inspection target area "guide sieve unit", and the aggregation unit 58 checks the total or average value of the inspection results of images 211, 221, and 231 by the preliminary inspection unit 57, and determines whether it is normal or abnormal depending on whether the average value or total value exceeds a predetermined threshold. This determination method is just one example, and for example, instead of the average value of the inspection results, a weighted determination may be made separately and a weighted average may be performed.

[0049] The display control unit 59 displays the multiple selected images selected by the image selection unit 53 and the inspection results from the inspection unit 56 on the display unit 61. In other words, the display control unit 59 displays the multiple selected images selected by the image selection unit 53 and the aggregation results from the aggregation unit 58 on the display unit 61. Specifically, the display control unit 59 generates text representing the aggregation results from the aggregation unit 58, and generates a video signal that arranges the text and selected images in the screen display area of ​​the display unit 61, and outputs the video signal to the display unit 61. The aggregation results from the aggregation unit 58 correspond to the inspection results from the inspection device 50.

[0050] Figure 5 shows the screen displayed on the display unit 61 after the inspection device 50 has performed an inspection based on the selected images shown in Figures 3(a) to (c). As shown in Figure 5, text 81 to 86 representing the summary results and selected images 91 to 94 are displayed on the display unit 61. Selected image 94 is an enlarged display of one of the selected images 91 to 93 selected by the user via the input unit 62.

[0051] Text 81 indicates that the inspection target area, the "guide sieve unit," is "normal" based on a majority vote of the inspection results of images 211, 221, and 231 from the preliminary inspection unit 57. The result "normal" indicates that the result is normal. Text 82 indicates that for the inspection target area "guide sieve unit," the number of selected images 211, 221, and 231 in which the inspection results by the preliminary inspection unit 57 are abnormal is "0," and the ratio of the number of selected images 211, 221, and 231 in which the inspection results are abnormal to the number of selected images selected by the image selection unit 53 is "0 abnormalities detected out of 3." The ratio of the number of selected images 211, 221, and 231 in which the inspection results are abnormal to the number of selected images selected by the image selection unit 53 can be determined from text 82. Text 83 indicates that the inspection result for the "boom" target area was "rust present," based on a majority vote of the inspection results of images 212,222,232 by the preliminary inspection unit 57. The result "rust present" indicates an abnormality. Text 84 indicates that for the inspection target area "boom," the number of selected images (212, 222, 232) whose inspection results by the preliminary inspection unit 57 are abnormal is "2," and the ratio of the number of selected images (212, 222, 232) whose inspection results are abnormal to the total number of selected images (212, 222, 232) selected by the image selection unit 53 is "2 out of 3 images detected as abnormal." The ratio of the number of selected images (212, 222, 232) whose inspection results are abnormal to the total number of selected images selected by the image selection unit 53 can be determined from text 84. Text 85 indicates that the inspection result for the "main winding rope" (the part to be inspected) is "kink present," based on a majority vote of the inspection results of images 213 and 223 by the preliminary inspection unit 57. The result "kink present" indicates an abnormality. Text 86 indicates that for the inspection target part "main winding rope," the number of selected images "2" in which the inspection results of images 213 and 223 by the preliminary inspection unit 57 are abnormal, and the ratio of the number of selected images in images 213 and 223 with abnormal inspection results to the number of selected images selected by the image selection unit 53 is "2 out of 3 images detected as abnormal." The ratio of the number of selected images in images 213 and 223 with abnormal inspection results to the number of selected images selected by the image selection unit 53 can be determined from text 86.

[0052] The display control unit 59 may also display the type of inspection selected by the inspection type selection unit 55 on the display unit 61, in addition to the multiple selected images and the summary results. Specifically, the display control unit 59 generates text representing the summary results from the summary unit 58 and text representing the type of inspection selected by the inspection type selection unit 55, and generates a video signal that places the text and selected images in the screen display area of ​​the display unit 61, and outputs the video signal to the display unit 61. Therefore, as shown in Figure 6, in addition to the texts 81-86 representing the summary results and the selected images 91-94, the texts 87-89 representing the type of inspection are displayed on the display unit 61.

[0053] Text 87 represents the type "object detection" applied to the inspection of the inspection target part "guide sieve unit". Text 88 represents the type "semantic domain segmentation" applied to the examination of the target area "boom". Text 89 represents the type "autoencoder" applied to the inspection of the inspection target part "main winding rope".

[0054] <4. Favorable Effects> As described above, the inspection system comprises an imaging unit 15 and an inspection device 50 that inspects the parts of the crane 100 to be inspected based on a plurality of selected images captured by the imaging unit 15. Therefore, the inspection of the parts of the crane 100 to be inspected can be performed efficiently.

[0055] Furthermore, the inspection device 50 has a display control unit 59 that displays the inspection results of the area to be inspected and the multiple selected images used for the inspection of the area to be inspected on the display unit 61. As a result, the user can easily grasp the inspection results of the area to be inspected and the multiple selected images used, and can judge the accuracy of the inspection. Thus, the labor involved in the inspection and related tasks is reduced.

[0056] Furthermore, the inspection device 50 includes an extraction / recognition unit 54, an inspection type selection unit 55, and a preliminary inspection unit 57. The extraction / recognition unit 54 extracts images of the target areas for inspection from each selected image and recognizes the type of target area for each image. The inspection type selection unit 55 selects the type of inspection according to the type of target area for each image of the target area in each selected image. The preliminary inspection unit 57 performs the selected type of inspection on each image of the target area based on the extracted images of the target areas. Therefore, the appropriate type of inspection according to the type of target area is performed on each target area. Consequently, the accuracy of the inspection is high.

[0057] Furthermore, the preliminary inspection unit 57 performs the inspection based on the image using a trained model that employs an algorithm of a type selected for the elephant according to the type of part to be inspected. The trained model is optimally designed so that the type of subcategory of inspection, the features, or both are matched to the type, location, range, or color of the part to be inspected. Therefore, the inspection accuracy is high.

[0058] <<Second Embodiment>> In the first embodiment described above, the image selection unit 53 selects multiple images. The extraction / recognition unit 54, the inspection type selection unit 55, and the preliminary inspection unit 57 then perform the above-described processing on each selected image, and the aggregation unit 58 aggregates the inspection results for each identical inspection target area.

[0059] Alternatively, the image selection unit 53 may select one image, and the extraction / recognition unit 54, the inspection type selection unit 55, and the preliminary inspection unit 57 may perform the above-described processing on that single selected image. In this case, the aggregation unit 58 does not perform aggregation processing.

[0060] Furthermore, the display control unit 59 displays a selected image chosen by the image selection unit 53 and the inspection results from the preliminary inspection unit 57 on the display unit 61. Specifically, the display control unit 59 generates text representing the aggregated results from the preliminary inspection unit 57, and generates a video signal that places the text and the selected image in the screen display area of ​​the display unit 61, and outputs the video signal to the display unit 61. The inspection results from the preliminary inspection unit 57 correspond to the inspection results from the inspection device 50.

[0061] Figure 7 shows the image displayed on the display unit 61 as a result of the inspection performed by the inspection device 50 based on the selected image shown in Figure 3(a). As shown in Figure 7, the text 181-183 representing the inspection result and the selected image 191 are displayed on the display unit 61.

[0062] Text 181 indicates that the inspection result of image 211 by the preliminary inspection unit 57 for the inspection target area "guide sieve unit" is "no abnormalities". An inspection result of "no abnormalities" indicates that the inspection is normal. Text 182 shows the inspection result "Rust present" from the preliminary inspection unit 57 of image 212 for the inspection target part "boom". The inspection result "Rust present" indicates an abnormality. Text 183 shows the inspection result "kink present" from the preliminary inspection unit 57 of image 213 for the inspection target part "main winding rope". The inspection result "kink present" indicates an abnormality. Note that only the inspection result is displayed here, but the numerical results that formed the basis of that judgment may also be displayed.

[0063] The display control unit 59 may also display the type of inspection selected by the inspection type selection unit 55 on the display unit 61, in addition to the multiple selected images and the summary results. In this case, as shown in Figure 8, the display unit 61 will display text 184 to 186 representing the type of inspection, in addition to the text 181 to 183 representing the summary results and the selected image 191.

[0064] Text 184 describes the type "object detection" applied to the inspection of the inspection target part "guide sieve unit". Text 185 represents the type "semantic domain segmentation" applied to the examination of the target area "boom". Text 186 describes the type of device, "autoencoder," applied to the inspection of the inspection target part, "main winding rope."

[0065] Except as described above, the second embodiment is the same as the first embodiment.

[0066] <<Third Embodiment>> In the third embodiment, the extraction and recognition unit 54 further extracts images (hereinafter referred to as "small images") for each color, material, or part from the images extracted for each part to be inspected. To explain with a specific example, in the case of the selected image shown in Figure 8, the extraction and recognition unit 54 extracts images 211, 212, and 213, and then further extracts small images 212a and 212b for each part from the image 212 of the detection target part "boom". Since the parts "support column" and "beam" have different colors, the small image 212a of the part "support column" and the small image 212b of the part "beam" are extracted by the extraction and recognition unit 54.

[0067] If a sub-image is extracted from the image of the area to be inspected, the inspection type selection unit 55 selects a sub-category such as inspection, algorithm, or machine learning training data for each sub-image in the image of the area to be inspected, according to the color, material, or component. Sub-categories selected for sub-images in the image of the same area to be inspected belong to the same major category.

[0068] If a sub-image is extracted from the image of the area to be inspected, the preliminary inspection unit 57 performs an inspection on the image of the area to be inspected, based on the sub-images extracted by the extraction and recognition unit 54, according to the sub-category selected by the inspection type selection unit 55. All trained models used for inspection are selected from a library in which the features are optimally designed to match the color, material, or parts in the sub-image.

[0069] The aggregation unit 58 aggregates the inspection results from the preliminary inspection unit 57 based on the image of the target area in each selected image, for each identical target area, as in the first embodiment. However, if a sub-image is extracted from the image of the target area, the aggregation unit 58 aggregates the inspection results from the preliminary inspection unit 57 based on the sub-image in each selected image for that target area.

[0070] Except as described above, the third embodiment is the same as the first embodiment.

[0071] <> Although embodiments have been described above, the present invention is not limited to the embodiments described above. In the embodiments described above, the imaging unit 15 mounted on the aircraft 10 was a so-called monocular camera, but it may also be a compound camera such as a stereo camera. Furthermore, the imaging unit 15 may be either a still camera or a video camera. In addition, the image sensor used for imaging may be one that emits wavelengths other than the so-called visible light region, such as ultraviolet or infrared light.

[0072] In the embodiments described above, a tower crane was given as an example of a crane that constitutes a structure, but it is not limited to this. For example, the structure includes all types of cranes, including mobile cranes such as crawler cranes, wheel cranes, and truck cranes, as well as port cranes, overhead cranes, gantry cranes, unloaders, and fixed cranes. Furthermore, the crane that constitutes a structure is not limited to cranes equipped with hooks, but may also be cranes that suspend attachments such as magnets and earth drill buckets. Moreover, the structure is not limited to cranes, but may also be heavy machinery such as construction equipment other than cranes, or structures (for example, steel towers).

[0073] In the embodiments described above, the computer 51 functions as an image storage unit 52, an image selection unit 53, an extraction / recognition unit 54, an inspection type selection unit 55, a preliminary inspection unit 57, an aggregation unit 58, and a display control unit 59, according to the computer program 64. In contrast, one or more second computers other than the computer 51 may individually function as at least one of the image storage unit 52, image selection unit 53, extraction / recognition unit 54, inspection type selection unit 55, preliminary inspection unit 57, aggregation unit 58, and display control unit 59, according to their respective computer programs, and the computer 51 and the second computers may cooperate. One of the one or more second computers may be mounted on the aircraft 10.

[0074] In the embodiments described above, the inspection by the inspection device 50, inspection unit 56, and preliminary inspection unit 57 determines whether there is an abnormality in the part to be inspected, the degree of abnormality, the degree of normality, or the likelihood. Alternatively, the inspection by the inspection device 50, inspection unit 56, and preliminary inspection unit 57 may determine the degree of the condition of the part to be inspected.

[0075] Furthermore, while each embodiment uses the system for the purpose of determining abnormalities in the parts of the structure to be inspected, any system that determines (inspects) the state of the target parts of the structure may be used. For example, a system that determines the degree of quality of an object may also be used. [Explanation of Symbols]

[0076] 10 flying objects 15 Imaging Unit 50 Inspection equipment 51 Computer 52 Image storage unit 53 Image Selection Section 54 Extraction / recognition section 55. Selection of examination type 56. Inspection Department 57 Preliminary Inspection Department 58. Aggregation Department 59 Display Control Unit 63 Storage medium 64 Computer Programs 100 Cranes (structures) 103 Boom (area to be examined) 105 Guide sieve unit (area to be inspected) 107 Main winding rope (part to be inspected)

Claims

1. A structural inspection system, An inspection unit that inspects the target part of the structure based on multiple images captured by the imaging unit, An extraction unit that extracts images of multiple subject areas for inspection from the multiple images, for each subject area for inspection, The system includes a display control unit that performs processing to display the results of the inspection of the subject area and the plurality of images used in the inspection of the subject area on a display unit. An inspection system in which the inspection unit performs an inspection of a type corresponding to the type of the part to be inspected, for each of the same parts to be inspected, based on the image of the part to be inspected extracted by the extraction unit.

2. The inspection system according to claim 1, wherein the display control unit causes the display unit to display the results of the inspection of the same inspection target part and the type of inspection for the inspection target part.

3. A structural inspection system, An inspection unit that inspects the target part of the structure based on multiple images captured by the imaging unit, The system includes a display control unit that performs processing to display the results of the inspection of the subject area and the plurality of images used in the inspection of the subject area on a display unit. The inspection unit performs multiple preliminary inspections based on the multiple images, and determines the final inspection result based on the results of the multiple preliminary inspections. The display control unit is an inspection system that displays the final inspection result on the display unit.

4. The inspection system according to claim 3, wherein the display control unit displays images on the display unit that show normal results from the preliminary inspection and images that show abnormal results from the preliminary inspection.

5. The inspection system according to claim 3 or 4, wherein the display control unit causes the display unit to display the percentage of the preliminary inspection results that were abnormal in a way that allows for identification.

6. A method for inspecting a structure, An inspection step in which the target part of the structure is inspected based on multiple images captured by the imaging unit, An extraction step of extracting images of multiple subject areas for examination from the multiple images, for each subject area; The system includes a display control step that performs processing to display the results of the inspection of the subject area and the plurality of images used for the inspection of the subject area on a display unit. A method for inspecting a structure, wherein in the inspection step, based on the image of the part to be inspected extracted in the extraction step, an inspection of a type corresponding to the type of part to be inspected is performed for each of the same parts to be inspected.

7. A method for inspecting a structure, An inspection step in which the target part of the structure is inspected based on multiple images captured by the imaging unit, The system includes a display control step that performs processing to display the results of the inspection of the subject area and the plurality of images used for the inspection of the subject area on a display unit. In the inspection step, multiple preliminary inspections are performed based on the multiple images, and the final inspection result is determined based on the results of the multiple preliminary inspections. A method for inspecting a structure in which the final inspection result is displayed on a display unit during the display control step.

8. An inspection step in which the target area of ​​a structure is inspected based on multiple images captured by the imaging unit, An extraction step of extracting images of multiple subject areas for examination from the multiple images, for each subject area; A computer program that causes a computer to perform a display control step of processing to display the results of the inspection of the subject area and the plurality of images used in the inspection of the subject area on a display unit, A computer program in which the computer performs, in the inspection step, an inspection of a type corresponding to the type of the part to be inspected, for each of the same parts to be inspected, based on the image of the part to be inspected extracted in the extraction step.

9. An inspection step in which the target area of ​​a structure is inspected based on multiple images captured by the imaging unit, A computer program that causes a computer to perform a display control step of processing to display the results of the inspection of the subject area and the plurality of images used in the inspection of the subject area on a display unit, The computer, in the inspection step, performs a plurality of preliminary inspections based on the plurality of images, and determines the final inspection result based on the results of the plurality of preliminary inspections. A computer program that causes the computer to display the final inspection result on the display unit in the display control step.

10. A storage medium storing the computer program described in claim 8 or 9.

Images