Device and method for cropping image of inner surface of piping, and program

Abstract

An image cropping device 1 includes an image development unit 12 for generating a plane development image by planarly developing a photographed image of an inner surface of piping, a measurement place designation unit 13 for designating, in the plane development image, a measurement part of a corrosion depth in the piping, and an image cropping unit 14 for cropping, from the plane development image, an image including the measurement part.

Description

Image extraction device and method for the inner surface of a pipe, and program

[0001] The present disclosure relates to an image extraction device, an image extraction method, and a program for an image of an inner surface of a pipe.

[0002] A huge number of lifeline facilities, such as water pipes and communication pipes, which are essential for daily life, are installed underground and are becoming increasingly dilapidated. Metal pipes in particular rust as they age, and as the rust progresses, holes can appear in the pipes, which can lead to road surface collapses due to the intrusion of soil and sand through the holes. Therefore, it is important to understand the depth of corrosion relative to the thickness of the pipes.

[0003] Previous research has examined strength assessment methods that take corrosion depth into account using ultrasonic measurements and numerical analysis to ensure structural safety (see Non-Patent Documents 1 and 2). However, these methods are difficult to apply to buried pipelines. On the other hand, a method using image recognition to detect abnormalities such as corrosion pitting in pipelines is known (see Non-Patent Document 3), and research is underway to establish a method to estimate corrosion depth from images of the inner surface of pipelines captured with a small camera. Estimating corrosion depth using image recognition requires measuring the corrosion depth and linking the depth to images of the measurement location for learning. Because pipeline wall thickness has tolerances and the cross section is circular, measuring it with conventional measuring tools is difficult. In recent years, research has been conducted to understand the condition of metals using scanning electron microscopes (SEMs) (see Non-Patent Documents 4 and 5), making it possible to measure corrosion depth using SEMs. Additionally, a method for cropping images using positional information within the image is known (see Non-Patent Document 6).

[0004] Tetsuhiro Shimosato and four others, "Research on Elucidation of Corrosion Deterioration Mechanism and Durability Diagnosis of Steel Bridges," Report on the Results of Research and Development of Technology Contributing to the Improvement of the Quality of Road Policy, No. 21-5, 2009. Kyosuke Inoue and six others, "Numerical Analysis of Compressive Load Carrying Capacity of Box Section Members of Steel Truss Bridges with Cross-Sectional Defects," Proceedings of the Structural Engineering Society, Vol. 65A, pp. 76-89, 2019. Srinath Shiv Kumar, et al., "Deep Learning-Based Automated Detection of Sewer Defects in CCTV Videos," Journal of Computing in Civil Engineering, Vol. 34, No. 1, 04019047. 2020 Masayasu Nagoshi, "Steel Plate Surface Analysis Technology Using Electron Microscopy," Materials and Environment, Vol. 64, No. 9, pp. 381-387, 2015 Tomoki Awano and two others, "Research on Image Recognition Technology Using Deep Learning in Metallographic Observation," IHI Technical Report, Vol. 60, No. 1, pp. 52-59, 2020 Keisuke Yoneda and two others, "Traffic Signal Recognition Considering False Detection Due to Multiple Lights," Transactions of the Society of Automotive Engineers of Japan, Vol. 48, No. 2, pp. 575-585, 2017

[0005] Conventional image recognition techniques for detecting defects (such as corrosion pits) in pipelines require measuring corrosion depth and associating it with an image (cropped image) containing the measurement point from an image of the pipeline interior. Because the area over which SEM can measure corrosion depth in a pipeline is extremely small, and the corresponding image area is also very small, images of only the measurement point are not suitable for deep learning. Therefore, it is necessary to generate a cropped image that captures an appropriate range of the pipeline wall. However, because a small camera captures the measurement point from an oblique angle, cropping an image using the method described in Non-Patent Document 6 results in differences in the range and resolution of the pipeline wall surface captured in the cropped image depending on the position of the measurement point, even if the image is cropped to the same size.

[0006] An image cutting-out device for the inner surface of a pipe in one embodiment includes an image development unit that flattens a captured image of the inner surface of a pipeline to generate a flattened image, a measurement point designation unit that designates a measurement point in the flattened image for measuring the corrosion depth in the pipeline, and an image cutting-out unit that cuts out an image including the measurement point from the flattened image.

[0007] A method for extracting an image of the inner surface of a pipe in one embodiment includes the steps of: planarly expanding a captured image of the inner surface of a pipeline to generate a planar expanded image; specifying a measurement point for the corrosion depth in the pipeline in the planar expanded image; and extracting an image including the measurement point from the planar expanded image.

[0008] A program according to one embodiment causes a computer to function as the image extraction device for the inner surface of a pipe.

[0009] According to the present disclosure, it is possible to identify the location in the captured image where the corrosion depth was measured, and to cut out the image in a pixel range suitable for deep learning.

[0010] 1 is a block diagram showing an example of the configuration of an image cutting device according to an embodiment; FIG. 2 is a flowchart showing an example of an image cutting method according to an embodiment; FIG. 3 is a diagram showing an example of a captured image; FIG. 4 is a diagram showing an example of processing by an image development unit in an image cutting device according to an embodiment; FIG. 5 is a diagram showing an example when the shape of an exclusion region is a circle; FIG. 6 is a diagram showing an example of eccentricity correction; FIG. 7 is an image showing an example of a cross section of a cut pipeline; FIG. 8 is an image showing an example of the inner surface of a pipeline as seen from the cutting position of the pipeline; FIG. 9 is a diagram showing an example of a cut surface (cutting position) in a pipeline and a measurement point for measuring the corrosion depth on the cut surface; and FIG. 10 is a diagram showing an example of designating a measurement point in a planar development image.

[0011] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. The present disclosure is not limited to the following embodiments, and various modifications can be made within the scope of the present disclosure.

[0012] <Image Cutout Device> Fig. 1 is a block diagram showing an example configuration of an image cutout device according to the present disclosure. As shown in Fig. 1, the image cutout device 1 includes an image information input unit 11, an image development unit 12, a measurement location designation unit 13, an image cutout unit 14, and a storage unit 15. The image information input unit 11, the image development unit 12, the measurement location designation unit 13, and the image cutout unit 14 form a control and arithmetic circuit (controller). The control and arithmetic circuit may be formed of dedicated hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array), or may be formed of a processor, or may include both.

[0013] The image information input unit 11 inputs an image (hereinafter referred to as "captured image") of the inner surface of a pipeline (inner surface of a pipe) and performance information of the camera that captured the inner surface of the pipeline. If the aspect ratio of the input captured image is not 1:1, the image information input unit 11 may resize the image to 1:1. The camera performance information includes the resolution and focal length of the camera.

[0014] The image development unit 12 develops the captured image on a plane, excluding the central region of the pipeline, to create a development of the inner surface of the pipeline (hereinafter referred to as a "developed plane image"). Details will be described later.

[0015] The measurement point designation unit 13 designates a point where the corrosion depth was measured (hereinafter referred to as the "measurement point") in the planar developed image created by the image development unit 12. The corrosion depth of the pipeline can be measured using an SEM. Details will be described later.

[0016] The image cropping unit 14 generates a cropped image by cropping an image including the measurement location designated by the measurement location designation unit 13. Details will be described later.

[0017] The storage unit 15 stores the cropped images generated by the image cropping unit 14. The storage unit 15 includes one or more memories, and may include, for example, a semiconductor memory, a magnetic memory, an optical memory, etc. Each memory included in the storage unit 15 may function, for example, as a main memory device, an auxiliary memory device, or a cache memory. The storage unit 15 stores any information used in the operation of the image cropping device 1. The storage unit 15 does not necessarily need to be provided inside the image cropping device 1, and may be provided outside the image cropping device 1.

[0018] <Image Extraction Method> Next, an example of the operation of the image extraction method according to the present disclosure will be described in more detail with reference to the flowchart shown in FIG. 2 and FIGS. 3 to 11 .

[0019] In step S101, the image information input unit 11 captures a captured image from a small camera capturing an image of the inside of a pipeline. FIG. 3 shows an example of a captured image. Image A is an image of the inner surface of a pipeline captured by a camera inserted into the pipeline, and the resolution decreases as the image approaches the area capturing the depth of the pipeline. The area capturing the center of the pipeline (hereinafter referred to as the "pipe center area") B has the lowest resolution. Furthermore, when cropped image P including measurement point p is compared with cropped image Q including measurement point q, cropped image P has high resolution and captures a small pipeline area, while cropped image Q has low resolution and captures a large pipeline area. Thus, differences in resolution and pipeline area depending on the position of the measurement point within the image present a problem.

[0020] 2 , in step S102, the image development unit 12 uses the camera's performance information to determine an area where the resolution is equal to or less than a threshold as an exclusion area. The shape of the exclusion area may be circular or rectangular. The image development unit 12 may identify the central area of ​​the pipeline using any known method such as a Hough transform or a depth map, and determine an exclusion area that includes the central area of ​​the pipeline.

[0021] In step S103, the image development unit 12 divides the captured image excluding the exclusion area by rectangular division, circular division, or the like, and performs planar transformation such as projective transformation or perspective projection transformation on each divided image.

[0022] FIG. 4 is a diagram illustrating an example of the processing performed by the image development unit 12. As shown in FIG. 4, the image development unit 12 divides the captured image A into divided images D to K, excluding an exclusion area C. The image development unit 12 then performs projective transformation on each of the divided images D to K to generate projectively transformed images D' to K'. The image development unit 12 may perform projective transformation using the number of pixels on the side of a rectangle that shows the closest part of the pipeline as a reference, so that the number of pixels on the opposite side matches that side. In other words, the image development unit 12 may perform projective transformation on the divided image D, using the upper base D1 as the side to be transformed and the lower base D2 as the reference side, so that the number of pixels on D1 and D2 matches. Note that the projectively transformed image D' in FIG. 4 is transformed by projective transformation. In this way, the image development unit 12 may appropriately resize the image. The image development unit 12 similarly performs projective transformation on the remaining divided images E to K to generate projectively transformed images E' to K'.

[0023] 5 is a diagram showing an example in which the shape of the exclusion area C is a circle. In this case, the image development unit 12 creates a planar development image L of the inner surface of the pipeline for an image obtained by excluding the exclusion area C from the largest circle M in the photographed image A (a circle inscribed on the edge of the photographed image A if the shape of the photographed image A is a square).

[0024] 2, in step S104, the image development unit 12 determines whether the exclusion area is decentered (not located at the center of the image). If the exclusion area is decentered, the process proceeds to step S105. If the exclusion area is not decentered, the process proceeds to step S106. Whether the exclusion area is decentered may be determined by comparing the center coordinates of the exclusion area with the center coordinates of the entire image.

[0025] In step S105, the image development unit 12 corrects the coordinates of the image division based on the amount of eccentricity of the exclusion area.

[0026] 6 is a diagram showing an example of decentering correction. In the example of FIG. 6, the central coordinate C of the exclusion area C of the photographed image A shown on the left side is ０ is the (xc, yc) at the center of the image, and the center coordinates C of the exclusion area C' of the photographed image A' shown on the right ０Since '(xc, yc') is decentered by yc'-yc from the center of the image in the y coordinate direction, the coordinates of the image division are decentered by yc'-yc.

[0027] Returning to Fig. 2, in step S106, all of the images segmented in steps S102 to S105 are projectively transformed, and the planarly transformed images are then joined together to create a planarly developed image of the inner surface of the pipeline. Referring to Fig. 4, the image development unit 12 combines projectively transformed images D' to K' of segmented images D to K to create a planarly developed image L of the inner surface of the pipeline. By converting an image of the inner surface of the pipeline, captured with a small camera or the like, into a planar image through image processing such as projective transformation, the inner surface of the pipeline captured in the image can be extracted at an equal distance. By planarizing an omnidirectional image of the inner surface of a deep structure such as a pipeline, the positional relationship of corrosion areas on the inner surface of the pipeline can be clarified.

[0028] Returning to FIG. 2 , in step S107, the measurement point designation unit 13 inputs information indicating a pipeline cross-sectional image, an inner pipeline image, or a measurement point of the corrosion depth, and designates a measurement point of the corrosion depth in the planar development image. To measure the corrosion depth using an SEM, the pipeline needs to be cut to a size that can be observed with the SEM. By recording the cut position of the pipeline in advance, it is possible to identify which cross section in the captured image corresponds to the cross section observed with the SEM. The pipeline cross-sectional image is an image showing a cross section of the pipeline cut for measuring the corrosion depth, as shown in FIG. 7 . The pipeline inner surface image is an image showing the inner surface of the pipeline as seen from the cut position of the pipeline, and includes the measurement point of the corrosion depth, as shown in FIG. 8 .

[0029] The measurement point designation unit 13 designates a point in the planarized image where the corrosion depth is measured based on a user's instruction. The user designates the measurement point based on the extent of corrosion within the pipeline, image features in the SEM image, etc. In this case, the accuracy of designating the measurement point can be improved by using image processing techniques such as feature point matching between the image of the inner surface of the pipeline and the planarized image.

[0030] FIG. 9 is a diagram showing a cross section (cut position) α of the pipeline 10 and a measurement point N of the corrosion depth at the cut position α. ​​FIG. 10 is a diagram showing an example of designating the measurement point N in the unfolded planar image L. The measurement point designation unit 13 identifies and records the cut position α in the axial direction of the pipeline 10. Next, the measurement point designation unit 13 identifies the measurement point N from information that can be used to identify the measurement point, such as characteristics of the spread of corrosion in the cut pipeline 10, such as rust bumps, and surrounding image information in the SEM image, and designates the measurement point N in the unfolded planar image L. The measurement point designation unit 13 may also identify the measurement point N from information indicating the measurement point. For example, if the information indicating the measurement point is 250 degrees, the measurement point designation unit 13 identifies the position corresponding to 250 degrees on the cut position α in the unfolded planar image L as the measurement point N, as shown in FIG. 10. By identifying the measurement point N of the corrosion depth on the inner surface of the pipeline, it is possible to crop the image centered on the measurement point N.

[0031] 2, in step S108, the image cropping unit 14 crops out a plurality of images from the planar development image within a predetermined pixel range, centered on the measurement point designated by the measurement point designation unit 13. In this case, the pixel range may be designated as a square or rectangular area, and this can be changed as desired.

[0032] FIG. 11 shows an example of image cropping. A pixel range is specified, for example, as a rectangular area, centered on the measurement location N specified in step S107, and the image is cropped within that pixel range. By cropping the image with different pixel counts, such as pattern a (100 x 100 pixels) and pattern b (200 x 200 pixels), as shown in FIG. 11, it is possible to compare the accuracy of deep learning depending on the size of the pixel range. Furthermore, it is possible to verify the effect on the accuracy of deep learning by expanding the pixel range circumferentially, such as pattern c (300 x 100 pixels). Alternatively, an arbitrary pattern, such as 150 x 150 pixels, can be specified. In some cases, cropping with this 150 x 150 pixel pattern is most suitable for deep learning. In this way, an arbitrary pixel range can be cropped around the corrosion depth measurement location, allowing the image to be cropped within a pixel range suitable for deep learning.

[0033] Returning to FIG. 2, in step S109 , the image cropping unit 14 stores the cropped image in the storage unit 15 .

[0034] As described above, in the present disclosure, the image captured by the small camera is converted into a plane and then unfolded into a plane, thereby identifying the location in the unfolded plane image where the corrosion depth was measured, and by cutting out the pipeline wall surface that appears in the image in an even range, it is possible to cut out the image in a pixel range suitable for deep learning.

[0035] <Program and Recording Medium> The above-described image extraction device 1 can be realized by a computer. A program may be provided to cause the image extraction device 1 to function as the image extraction device 1. The program may be stored in a storage medium or provided via a network.

[0036] 12 is a block diagram showing a schematic configuration of a computer 100 functioning as the image extraction device 1. Here, the computer 100 may be a general-purpose computer, a dedicated computer, a workstation, a personal computer (PC), an electronic notepad, etc. The program instructions may be program code, code segments, etc. for performing the necessary tasks.

[0037] 12, the computer 100 includes a processor 101, a read-only memory (ROM) 102, a random access memory (RAM) 103, a storage 104, an input unit 105, an output unit 106, and a communication interface 107. Each component is connected to each other via a bus 108 so as to be able to communicate with each other. The processor 101 is specifically a central processing unit (CPU), a micro processing unit (MPU), a graphics processing unit (GPU), a digital signal processor (DSP), a system on a chip (SoC), or the like, and may be configured by multiple processors of the same type or different types.

[0038] The processor 101 controls each component and executes various arithmetic processes. That is, the processor 101 reads a program from the ROM 102 or the storage 104 and executes the program using the RAM 103 as a work area. The processor 101 controls each component and executes various arithmetic processes in accordance with the program stored in the ROM 102 or the storage 104. In the above-described embodiment, the program according to the present disclosure is stored in the ROM 102 or the storage 104.

[0039] The program may be stored in a storage medium readable by the computer 100. Using such a storage medium, the program can be installed on the computer 100. Here, the storage medium on which the program is stored may be a non-transitory storage medium. The non-transitory storage medium is not particularly limited, and may be, for example, a CD-ROM, a DVD-ROM, or a USB (Universal Serial Bus) memory. Furthermore, the program may be downloaded from an external device via a network.

[0040] The ROM 102 stores various programs and various data. The RAM 103 temporarily stores programs or data as a working area. The storage 104 is configured with an HDD (Hard Disk Drive) or an SSD (Solid State Drive) and stores various programs including an operating system and various data.

[0041] The input unit 105 includes one or more input interfaces that receive input operations from a user and acquire information based on the user operations. For example, the input unit 105 may be, but is not limited to, a pointing device, a keyboard, a mouse, etc.

[0042] The output unit 106 includes one or more output interfaces that output information. For example, the output unit 106 is a display that outputs information as a video or a speaker that outputs information as an audio, but is not limited to these. Note that if the output unit 106 is a touch panel display, it also functions as the input unit 105.

[0043] The communication interface 107 is an interface for communicating with an external device.

[0044] The following additional notes are provided regarding the above-described embodiments.

[0045] (Supplementary Item 1) An image cropping device for the inner surface of a pipe, comprising: a memory; and a controller connected to the memory, wherein the controller: plane-develops a captured image of the inner surface of a pipe to generate an unfolded image; specifies a measurement location in the unfolded image for measuring corrosion depth in the pipe; and crops out an image including the measurement location from the unfolded image. (Supplementary Item 2) The image cropping device for the inner surface of a pipe according to Supplementary Item 1, wherein the image development unit divides the captured image excluding a central area of ​​the pipe where the center of the pipe is captured, and then combines the divided images by plane transformation to generate the unfolded image. (Supplementary Item 3) The image cropping device for the inner surface of a pipe according to Supplementary Item 2, wherein the image development unit identifies the central area of ​​the pipe, which has low resolution, in the captured image based on performance information of a camera that photographed the pipe. (Supplementary Item 4) The image cropping device according to any one of Supplementary Items 1 to 3, wherein the measurement location designation unit designates a cut position of the pipe and the measurement location from an image measuring the corrosion depth at the cut position. (Supplementary Item 5) The image cropping device for the inner surface of a pipe according to Supplementary Item 4, wherein the measurement point designation unit designates the measurement point by image feature point matching between an image of the inner surface of the pipe showing the inner surface of the pipe as seen from a cut position of the pipe and the unfolded image. (Supplementary Item 6) The image cropping device for the inner surface of a pipe according to any one of Supplementary Items 1 to 5, wherein the image cropping unit crops out a plurality of images centered on the measurement point from the unfolded image. (Supplementary Item 7) A method for cropping an image of the inner surface of a pipe, comprising: developing a captured image of the inner surface of a pipe to generate an unfolded image; designating a measurement point in the unfolded image for measuring the corrosion depth in the pipe; and cropping an image including the measurement point from the unfolded image. (Supplementary Item 8) A non-transitory storage medium storing a program executable by a computer, the non-transitory storage medium storing a program causing the computer to function as the image cropping device for the inner surface of a pipe according to any one of Supplementary Items 1 to 7.

[0046] Although the above-described embodiments have been described as typical examples, it will be apparent to those skilled in the art that many modifications and substitutions can be made within the spirit and scope of the present disclosure. Therefore, the present invention should not be interpreted as being limited by the above-described embodiments, and various modifications or alterations are possible without departing from the scope of the claims. For example, multiple building blocks shown in the block diagrams of the embodiments can be combined into one, or one building block can be divided.

[0047] REFERENCE SIGNS LIST 1 Image extraction device 10 Pipe 11 Image information input unit 12 Image development unit 13 Measurement point designation unit 14 Image extraction unit 15 Memory unit 100 Computer 101 Processor 102 ROM 103 RAM 104 Storage 105 Input unit 106 Output unit 107 Communication interface 108 Bus

Claims

1. An image extraction device for the inner surface of a pipe, comprising: an image expansion unit that expands a captured image of the inner surface of a pipe to generate an expanded image; a measurement location designation unit that designates a measurement location in the expanded image for measuring the depth of corrosion in the pipe; and an image extraction unit that cuts out an image including the measurement location from the expanded image.

2. The image extraction device for the inner surface of a pipe as described in claim 1, wherein the image expansion unit divides the captured image excluding a central area of ​​the pipe in which the center of the pipe is captured, and then performs planar transformation on each of the divided images to combine them to generate the planar expansion image.

3. The image extraction device for the inner surface of a pipe as described in claim 2, wherein the image expansion unit identifies the central area of ​​the pipe, which has low resolution in the captured image, based on performance information of a camera that photographed the pipe.

4. The image extraction device for the inner surface of a pipe according to claim 1, wherein the measurement point designation unit designates the measurement point from a cut position of the pipe and a measurement image of the corrosion depth at the cut position.

5. An image extraction device for the inner surface of a pipe as described in claim 4, wherein the measurement point designation unit designates the measurement point by matching image feature points between an image of the inner surface of the pipe showing the inner surface of the pipe as seen from the cut position of the pipe and the planar development image.

6. The image extraction device for the inner surface of a pipe as described in claim 1, wherein the image extraction unit extracts a plurality of images from the planar development image, the images being centered on the measurement point.

7. A method for extracting an image of the inner surface of a pipe, comprising the steps of: planarly expanding a captured image of the inner surface of a pipe to generate a planar expanded image; specifying a measurement point for measuring the corrosion depth in the pipe in the planar expanded image; and extracting an image including the measurement point from the planar expanded image.

8. A program for causing a computer to function as the image extraction device for the inner surface of a pipe according to claim 1.

Images