Deterioration detection system, degradation detection method, program, region image correction device, and machine learning model for region correction

Abstract

To improve extracting accuracy of a region corresponding to a structure.SOLUTION: A deterioration detection system comprises: an acquisition unit which acquires a captured image including a structure; a generation unit which generates a structure region image which covers the structure included in the captured image; a correction unit which corrects the structure region image; and a detection unit which detects deterioration of the structure on the basis of the structure region image corrected by the correction unit. The correction unit inputs the structure region image generated by the generation unit to a machine learning model for region correction, and outputs the result obtained by correcting the structure region image from the machine learning model for region correction.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a deterioration detection system, a deterioration detection method, a program, an area image correction device, and a machine learning model for area correction.

Background Art

[0002] Conventionally, a technique for determining the degree of deterioration of a structure using an image of a structure such as a steel tower or a bridge has been known. This type of technique is described in, for example, Patent Document 1. The information processing method described in Patent Document 1 performs an abnormality diagnosis by analyzing an image of power distribution facilities using a machine learning model. This information processing method first outputs the type of diagnostic target device included in the image, the area in the image, and the estimated accuracy using a neural network, and then analyzes the image area where the diagnostic target device appears using another neural network to perform an abnormality diagnosis of the diagnostic target device.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In conventional techniques such as the information processing method described in Patent Document 1 mentioned above, the area of the diagnostic target device in the image is extracted, but due to overlooking where the structure is classified as the background, there are cases where what should originally be a structure is excluded from the objects for determining the degree of deterioration. Also, in conventional techniques, the accuracy of determining the degree of deterioration has been reduced due to false detection where a part of the background such as electric wires or tree branches is classified as a structure.

[0005] The present disclosure has been made in view of such circumstances, and an object thereof is to provide a deterioration detection system, a deterioration detection method, a program, an area image correction device, and a machine learning model for area correction that can improve the extraction accuracy of an area corresponding to a structure.

Means for Solving the Problems

[0006] The present disclosure has been made to solve the above-described problems. One aspect of the present disclosure is a deterioration detection system including an acquisition unit that acquires a captured image including a structure, a generation unit that generates a structure area image covering the structure included in the captured image, a correction unit that corrects the structure area image, and a detection unit that detects deterioration of the structure based on the structure area image corrected by the correction unit.

[0007] Another aspect of the present disclosure is a deterioration detection method for causing a deterioration detection system to execute a step of acquiring a captured image including a structure, a step of generating a structure area image covering the structure included in the captured image, a step of correcting the structure area image, and a step of detecting deterioration of the structure based on the corrected structure area image.

[0008] Another aspect of the present disclosure is a program for causing a computer of an information processing apparatus to execute a step of acquiring a captured image including a structure, a step of generating a structure area image covering the structure included in the captured image, a step of correcting the structure area image, and a step of detecting deterioration of the structure based on the corrected structure area image.

[0009] Another aspect of the present disclosure is an area correction device that inputs a structure area image covering a structure included in a captured image output from a machine learning model for area extraction to a machine learning model for area correction learned by an annotation image obtained by dividing a captured image for learning into a structure part and a background part and a structure area image obtained by extracting an area representing the structure from the captured image, and corrects the structure area image based on the output from the machine learning model for area correction.

[0010] Another aspect of the present disclosure is a machine learning model for area correction that is learned by an annotation image obtained by dividing a captured image for learning into a structure part and a background part and a structure area image obtained by extracting an area representing the structure from the captured image, and outputs a result of correcting the structure area image when a structure area image covering a structure included in the captured image output from the machine learning model for area extraction is input.

Advantages of the Invention

[0011] According to one aspect of the present invention, the extraction accuracy of the area corresponding to the structure can be increased.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Embodiments for Carrying Out the Invention

[0013] Hereinafter, a deterioration detection system, a deterioration detection method, a program, a region image correction device, and a machine learning model for region correction to which the present invention is applied will be described with reference to the drawings.

[0014] (First Embodiment) FIG. 1 is a block diagram showing an example of a deterioration detection system 1 in the first embodiment. The deterioration detection system 1 detects the deterioration of a structure using a captured image obtained by photographing the structure, such as a steel tower, with a camera device mounted on, for example, a drone. In the embodiment, a steel tower is described as the structure, but it is not limited thereto, and the structure may be any structure that needs to detect the deterioration of power facilities such as utility poles and electric wires, and infrastructure facilities such as bridges.

[0015] The deterioration detection system 1 includes, for example, a deterioration detection device 100 and a terminal device 200. The deterioration detection device 100 and the terminal device 200 are connected to a communication network. The communication network may include, for example, a general-purpose network such as the Internet, and a private network such as local 5G or WiFi (registered trademark). The deterioration detection device 100 and the terminal device 200 have a communication interface (not shown) such as a NIC (Network Interface Card) or a wireless communication module for connecting to a network such as the Internet.

[0016] The terminal device 200 is an information processing device operated by, for example, an administrator who manages a structure. The terminal device 200 transmits a captured image of the structure captured by the camera device to the deterioration detection device 100. As a response to the transmitted captured image, the terminal device 200 receives a deterioration detection result from the deterioration detection device 100. The terminal device 200 performs processes such as display based on the deterioration detection result.

[0017] The deterioration detection device 100 is an information processing device that communicates with other devices such as the terminal device 200 and performs various processes. The deterioration detection device 100 includes, for example, an acquisition unit 110, a generation unit 120, a proofreading unit 130, a detection unit 140, and an output unit 150. The acquisition unit 110, the generation unit 120, the proofreading unit 130, the detection unit 140, and the output unit 150 are realized, for example, by a computer such as a CPU (Central Processing Unit) executing a program stored in a program memory. In the embodiment, the acquisition unit 110, the generation unit 120, the proofreading unit 130, the detection unit 140, and the output unit 150 are provided in the deterioration detection device 100, but they may be distributed and arranged in a plurality of devices.

[0018] The acquisition unit 110 acquires a captured image including a structure. The generation unit 120 generates a structure region image that covers the structure included in the captured image. The structure region image is a mask image formed by coordinates recognized as a structure. The process of generating the structure region image is, for example, a process of removing the background of a structure such as a tower from the captured image, and a process of extracting only the region of the tower from the captured image and generating coordinate information covering the structure. The structure region image is, for example, an image in which the value of a pixel including a structure is set to "1" and the value of a pixel not including a structure is set to "0". The generation unit 120 generates a structure region image by rule-based processing using existing image processing techniques, but may use a machine learning model as described later, and may use a combination of rule-based processing and processing using a machine learning model.

[0019] The correction unit 130 corrects the structural area image. The process of correcting the structural area image is a process for correcting the structural area image into an image corresponding to the actual structure. The process of correcting the structural area image includes, for example, a process of complementing the missing part of the structural area image and a process of deleting the image in the structural area image that does not correspond to the structure. The image in the structural area image that does not correspond to the structure is a misrecognition area erroneously determined by the generation unit 120 to have a structure. The correction unit 130 corrects the structural area image by rule-based processing using existing image processing techniques, but may use a machine learning model as described later, and may use both rule-based processing and processing using a machine learning model in combination.

[0020] The detection unit 140 detects the deterioration of the structure based on the structural area image corrected by the correction unit 130. The deterioration of the structure is, for example, a state in which maintenance of the structure is required, such as rust, damage, or loosening of screws of the structure. The detection unit 140 divides, for example, the captured image with the structural area image superimposed into a plurality of patch images, and detects the deterioration based on the pixel information of the area in the patch image where the structural area image is superimposed. The process of detecting the deterioration includes a process of determining the presence or absence of deterioration and a process of calculating the degree of deterioration in the area where deterioration is determined to be present. The detection unit 140 detects the deterioration by rule-based processing using existing image processing techniques, but may use a machine learning model as described later, and may use both rule-based processing and processing using a machine learning model in combination.

[0021] The output unit 150 outputs the result of detecting the deterioration by the detection unit 140.

[0022] FIG. 2 is a flowchart showing an example of the operation procedure of the deterioration detection device 100 in the first embodiment. First, the acquisition unit 110 acquires a captured image from the terminal device 200 (step S100). Next, the generation unit 120 extracts a region covering the structure from the acquired captured image and generates a structure region image (step S102). Next, the proofreading unit 130 proofreads the structure region image generated by the generation unit 120 (step S104). Next, the detection unit 140 detects the deterioration of the structure using the image corresponding to the proofread structure region image in the captured image (step S106). Next, the output unit 150 outputs the deterioration detection result detected by the detection unit 140 to the terminal device 200 (step S108).

[0023] FIG. 3 is a diagram showing an example of a captured image in the embodiment. The captured image includes, for example, a background image, a tower portion image, and a deteriorated portion image. FIG. 4 is an example of a structure region image in the embodiment. The structure region image includes a structure region image corresponding to the tower portion image. The structure region image is an image that would originally be generated in the region covering the tower portion image and is not generated in the region corresponding to the background image. However, the structure region image generated by the generation unit 120 includes a missing portion where a part of the region corresponding to the structure is missing and a misrecognition region generated in the region where there is no structure. FIG. 5 is an example of the proofread structure region image in the embodiment. The structure region image shown in FIG. 4 is corrected to a proofread structure region image in which the missing portion is complemented and the misrecognition region is deleted by being proofread by the proofreading unit 130. FIG. 6 is a diagram showing an example of an image extracted from the captured image using the proofread structure region image in the embodiment. The detection unit 140 superimposes the proofread structure region image proofread by the proofreading unit 130 on the captured image, extracts the captured image at the superimposed location, and sets the locations where the proofread structure region image is not superimposed to black without extraction, thereby obtaining a captured image as shown in FIG. 6. FIG. 7 is a diagram showing an example of the deterioration detection result in the embodiment. The detection unit 140 divides the captured image shown in FIG. 6 into a plurality of patch images and detects deterioration for each patch image. The detection unit 140 detects that the degree of deterioration of a patch image without deterioration is "0", and detects the degree of deterioration and the coordinates of the deterioration region of a patch image with deterioration. The degree of deterioration may be higher as the area where deterioration is detected is larger, but is not limited thereto. Such a deterioration detection result is transmitted to the terminal device 200 by the output unit 150 and displayed on the terminal device 200.

[0024] As described above, according to the deterioration detection system 1 of the first embodiment, the structure region image covering the structure included in the captured image including the structure is checked, and the deterioration of the structure is detected based on the checked structure region image. Therefore, the extraction accuracy of the region corresponding to the structure can be increased, and the deterioration of the structure can be detected with high accuracy.

[0025] (Second Embodiment) FIG. 8 is a block diagram showing an example of the deterioration detection system 1A in the second embodiment. The deterioration detection system 1A is different from the above-described deterioration detection system 1 in that it includes a generation unit 120A that uses a region extraction model 122, a checking unit 130A that uses a region checking model 132, and a detection unit 140A that uses a deterioration detection model 142. The generation unit 120A, the checking unit 130A, and the detection unit 140A function as an inference unit 102 that makes inferences using a machine learning model. The deterioration detection system 1A may include a learning device 300. The learning device 300 learns the region extraction model 122, the region checking model 132, and the deterioration detection model 142 by performing learning processing on the machine learning model.

[0026] The region extraction model 122 is realized by a machine learning model for region extraction such as, for example, a neural network or deep learning. Specifically, the region extraction model 122 may be realized by PspNet, but is not limited thereto. The region extraction model 122 is trained using as training data a first training captured image (patch image) and an annotation image obtained by dividing the first training captured image into a structure part and a background part. The generation unit 120A inputs the captured image obtained by the acquisition unit 110 to the region extraction model 122, and generates a structure region image based on the output of the region extraction model 122.

[0027] The region refinement model 132 is realized by a machine learning model for region refinement such as, for example, an adversarial generative network (GAN), but is not limited thereto. The region refinement model 132 is trained using an annotation image obtained by dividing a second training captured image into a structure part and a background part, and a structure region image obtained by extracting a region representing a structure from the second training captured image output from the machine learning model for region extraction. When a structure region image covering the structure included in the captured image output from the region extraction model 122 is input to the region refinement model 132, the refinement unit 130A outputs a result obtained by refining the structure region image from the region refinement model 132.

[0028] The deterioration detection model 142 is realized by, for example, a machine learning model for deterioration detection such as a neural network like an AutoEncoder, but is not limited thereto. The deterioration detection model 142 is trained using, as training images, an image obtained by applying an annotation image divided into a structure part and a background part as a mask to a first training captured image, and an image obtained by applying an annotation image divided into a structure part and a background part as a mask to a second training captured image. The detection unit 140A inputs the refined structure region image output from the region refinement model 132 and the captured image acquired by the acquisition unit 110 to the deterioration detection model 142, and detects the deterioration of the structure based on the output from the deterioration detection model 142. For example, an AutoEncoder may be prepared for each type of deterioration, and the detection unit 140A may create a deterioration detection result corresponding to the type of deterioration based on the AutoEncoder that detected the deterioration.

[0029] The deterioration detection model 142 may use different machine learning models depending on the type of structure. The type of structure is, for example, a steel tower, a utility pole, an electric wire, etc., and the deterioration detection model 142 may be different machine learning models for a steel tower, a utility pole, and an electric wire. Also, the type of structure may be the painting of a steel tower, and the deterioration detection model 142 may be different machine learning models depending on the type (color) of the painting of the steel tower.

[0030] The deterioration detection system 1A may be configured such that at least the refinement unit 130A uses the region refinement model 132, the generation unit 120A does not use the region extraction model 122, and the detection unit 140A does not use the deterioration detection model 142. The refinement unit 130A functions as a region image refinement device that inputs the structure region image generated by the generation unit 120A to the region refinement model 132 and outputs the result of refining the structure region image from the region refinement model 132.

[0031] FIG. 9 is a block diagram showing an example of the learning device 300 in the second embodiment. The learning device 300 is connected to the storage device 400 via, for example, a communication network. The learning device 300 acquires data necessary for learning from the storage device 400. The learning device 300 is an information processing device that executes, for example, a learning program P10, a learning program P12, and a learning program P14 by a processor such as a CPU.

[0032] The learning program P10 acquires, as learning data, data D10 including a patch image set a and an annotation image of the patch image set a. The patch image set a is a set of captured images including, for example, a steel tower, and the annotation image of the patch image set a is an image in which regions in the patch images in each of the patch image sets a are manually tagged as a steel tower part and a background part. The learning program P10 outputs one of the patch image sets a to the region extraction model 122, and compares the inference result of the region extraction output from the region extraction model 122 with an annotation image of one of the patch image sets a. The inference result of the region extraction is information indicating whether the region in the patch image is a steel tower region or a background region. The learning program P10 learns the region extraction model 122 so that the inference result of the region extraction approaches an annotation image of one of the patch image sets a.

[0033] The region extraction model 122 acquires the patch image set b as the data D12 to be inferred. The region extraction model 122 outputs the structure region image (D16) as the inference result for each of the patch image sets b. The structure region image (D16) is input to the learning program P12. The annotation image (D14) of the patch image set a in which the patch images in each of the patch image sets b are manually tagged as the tower part and the background part is input to the learning program P12. The learning program P12 outputs one of the structure region images for each of the patch image sets b to the region refinement model 132, and compares the inference result of the region refinement output from the region refinement model 132 with one annotation image of the patch image set b. The inference result of the region refinement is information indicating whether the region in the patch image is the tower region or the background region. The learning program P12 trains the region refinement model 132 so that the inference result of the region refinement approaches one annotation image of the patch image set a.

[0034] The learning program P14 acquires the image obtained by masking the patch image set a with the annotation image, the image obtained by masking the patch image set b with the annotation image, the patch image set b, and the patch image label b as learning data. The patch image label is information indicating the presence or absence of degradation in the patch image. The learning program P14 inputs the patch images of the patch image set a to the degradation detection model 142, and trains the degradation detection model 142 so that the degradation detection result output from the degradation detection model 142 approaches the patch image label a. The learning program P14 inputs the patch images of the patch image set b to the degradation detection model 142, and trains the degradation detection model 142 so that the degradation detection result output from the degradation detection model 142 approaches the patch image label b.

[0035] FIG. 10 is a block diagram showing an example of the inference unit 102 in the second embodiment. The inference unit 102 is connected to the storage device 500 via, for example, a communication network. The inference unit 102 acquires data necessary for inference from the storage device 500. The inference unit 102 is an information processing device that executes, for example, the region extraction model 122, the region correction model 132, and the degradation detection model 142 by a processor such as a CPU.

[0036] The inference unit 102 acquires a patch image (D20) obtained by dividing an imaging image of a structure to be inferred and inputs it to the region extraction model 122. The region extraction model 122 outputs a structure region image (D22) in response to the input of the patch image (D20). The region correction model 132 inputs the structure region image (D22) output from the region extraction model 122. The region correction model 132 outputs a corrected structure region image (D24) in response to the input of the structure region image (D22). The inference unit 102 inputs a patch image (D26) including only the structure region overlapping with the structure region image (D24) among the patch images (D20) to the degradation detection model 142. The degradation detection model 142 outputs a patch image label indicating the presence or absence of degradation in response to the input of the patch image (D26). The degradation detection model 142 may infer the degradation level and the type of degradation in addition to the presence or absence of degradation. The degradation detection model 142 may output information indicating the coordinates of the region with degradation.

[0037] According to the degradation detection system 1A of the second embodiment, by inputting a structure region image to a machine learning model for region correction and outputting the result of correcting the structure region image from the machine learning model for region correction, the extraction accuracy of the region corresponding to the structure can be increased.

[0038] Although the embodiments and modifications have been described, they are merely examples and are not limited thereto. For example, any one of the embodiments or modifications, or a part of each embodiment or a part of each modification may be combined with one or more other embodiments or one or more other modifications to realize an aspect of the present invention.

Description of Reference Numerals

[0039] 1, 1A... Degradation detection system, 100... Degradation detection device, 102... Inference unit, 110... Acquisition unit, 120, 120A... Generation unit, 122... Region extraction model, 130, 130A... Proofreading unit, 132... Region proofreading model, 140, 140A... Detection unit, 142... Degradation detection model, 150... Output unit, 200... Terminal device, 300... Learning device, 400, 500... Storage device

Claims

1. An acquisition unit that acquires a captured image including a structure; A generation unit that generates a structure region image covering the structure included in the captured image; An annotation unit that annotates the structure region image; A detection unit that detects degradation of the structure based on the structure region image annotated by the annotation unit; A degradation detection system comprising the above.

2. The annotation unit inputs the structure region image generated by the generation unit into a machine learning model for region annotation, and outputs the result of annotating the structure region image from the machine learning model for region annotation. The degradation detection system according to Claim 1.

3. The generation unit inputs the captured image acquired by the acquisition unit into a machine learning model for region extraction that has been learned using a first learning captured image and an annotation image obtained by dividing the first learning captured image into a structure part and a background part as learning data, and generates the structure region image based on the output of the machine learning model for region extraction. When the structure region image covering the structure included in the captured image output from the machine learning model for region extraction is input to a machine learning model for region annotation that has been learned using an annotation image obtained by dividing a second learning captured image into a structure part and a background part and a structure region image obtained by extracting the region representing the structure from the second learning captured image output from the machine learning model for region extraction, the annotation unit outputs the result of annotating the structure region image from the machine learning model for region annotation. The degradation detection system according to Claim 1.

4. The detection unit inputs the annotated structure region image output from the machine learning model for region extraction and the captured image acquired by the acquisition unit into a machine learning model for degradation detection that has been learned using an image obtained by masking the first learning captured image with an annotation image obtained by dividing it into a structure part and a background part and an image obtained by masking the second learning captured image with an annotation image obtained by dividing it into a structure part and a background part as learning images, and detects degradation of the structure based on the output from the machine learning model for degradation detection. The degradation detection system according to Claim 3.

5. A step in which the degradation detection system acquires a captured image including a structure; A step in which the degradation detection system generates a structure region image covering the structure included in the captured image; A step in which the degradation detection system annotates the structure region image; ​ ​ The step of the deterioration detection system detecting the deterioration of the structure based on the corrected structure region image; A deterioration detection method for causing the above to be executed.

6. On the computer of the information processing apparatus, The step of acquiring a captured image including a structure; The step of generating a structure region image covering the structure included in the captured image; The step of proofreading the structure region image; The step of detecting the deterioration of the structure based on the corrected structure region image; A program for causing the above to be executed.

7. An area proofreading machine learning model learned by an annotation image obtained by dividing a learning captured image into a structure part and a background part, and a structure region image obtained by extracting a region representing a structure from the captured image output from a region extraction machine learning model. When a structure region image covering the structure included in the captured image output from the region extraction machine learning model is input, the structure region image is proofread based on the output from the area proofreading machine learning model. An area image proofreading device.

8. Learned by an annotation image obtained by dividing a learning captured image into a structure part and a background part, and a structure region image obtained by extracting a region representing a structure from the captured image output from a region extraction machine learning model, When a structure region image covering the structure included in the captured image output from the region extraction machine learning model is input, the result of proofreading the structure region image is output. An area proofreading machine learning model.

Images