Detection device

Abstract

A detection device (10) comprises: a control unit (14) that, from a captured image of a structure, predicts the state of a concrete surface of the structure before deterioration, identifies an image region in the captured image having a difference from the surface image obtained as a result of the prediction, detects a predetermined deterioration of the concrete surface in the image region having the identified difference, and generates a display image that distinguishes, in the captured image, a first image region in which the predetermined deterioration was detected and a second image region in which the predetermined deterioration was not detected; and an output unit (13) that outputs the display image generated by the control unit (14) to a display device (30).

Description

Detection device

[0001] The present disclosure relates to a detection device.

[0002] Conventionally, techniques for inspecting concrete structures have been known. Concrete structures such as roads, tunnels, dams, and bridges are important social infrastructure facilities that support Japan's socio-economic activities and the lives of its people. Managers of these facilities have dispatched inspectors to their sites to conduct regular visual inspections in order to safely maintain and manage the facilities.

[0003] Furthermore, with the recent remarkable progress in digital technology, the introduction of technology for performing inspections based on images of structures taken with digital cameras is being promoted, replacing on-site visual inspections by inspectors. For example, Non-Patent Document 1 discloses a technology for detecting cracks, fissures, or exposed rebar using image recognition in order to improve the efficiency of inspectors' visual inspections of deterioration from photographed images of the concrete surface of a structure.

[0004] Yusuke Fujita and two others, "High-precision automatic crack extraction in concrete structures using image processing," Journal of the Japan Society of Civil Engineers, Vol. 66, No. 3, pp. 459-470, September 2010.

[0005] The technology disclosed in Non-Patent Document 1 does not detect small abnormalities on the concrete surface, such as wear on the concrete surface, spalling where parts have peeled off, or junk where aggregate is densely packed, which are difficult to detect directly from captured images.

[0006] The purpose of the present disclosure, made in consideration of the above circumstances, is to detect small abnormalities on the concrete surface of a structure.

[0007] A detection device according to one embodiment of the present disclosure includes a control unit that predicts the state of the concrete surface of a structure before deterioration from a photographed image of the structure, identifies an image area in the photographed image that differs from the surface image obtained as a result of the prediction, detects specified deterioration of the concrete surface in the identified image area with the difference, and generates a display image in which a first image area in which the specified deterioration is detected and a second image area in which the specified deterioration is not detected are distinguished on the photographed image; and an output unit that outputs the display image generated by the control unit to a display device.

[0008] According to one embodiment of the present disclosure, it is possible to detect small abnormalities on the concrete surface of a structure.

[0009] FIG. 1 is a block diagram illustrating a schematic configuration example of a detection device according to an embodiment of the present disclosure; FIG. 2 is a diagram illustrating a prediction function of the detection device; FIG. 3 is a diagram illustrating a detection function of the detection device; FIG. 4 is a flowchart illustrating an operation example of the detection device according to an embodiment of the present disclosure; FIG. 5 is a diagram illustrating an example of a captured image and a surface image; FIG. 6 is a diagram illustrating division into rectangular regions; FIG. 7 is a diagram illustrating a result of identifying an image region having a difference; FIG. 8 is a diagram illustrating an example of a display image; FIG. 9 is a block diagram illustrating a schematic configuration example of a computer functioning as a detection device;

[0010] (Overview of the embodiment) An overview of a system 1 according to an embodiment of the present disclosure will be described with reference to Fig. 1. The system 1 includes a detection device 10, an imaging device 20, and a display device 30.

[0011] The detection device 10 is a computer that inspects the surface condition of concrete structures such as roads, tunnels, dams, and bridges.

[0012] The photographing device 20 is a digital camera that photographs a structure. The photographing device 20 outputs a photographed image p1 of the structure as a moving image or a still image to the detection device 10.

[0013] The display device 30 includes at least one display interface capable of displaying image data. The display interface is, for example, a display. The display is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescent) display. However, the display interface is not limited to these. The display device 30 displays the image output from the detection device 10 on a display screen.

[0014] Next, the configuration of the detection device 10 will be described in detail.

[0015] (Configuration of the Detection Device) The detection device 10 includes an input unit 11, a storage unit 12, an output unit 13, and a control unit 14.

[0016] The input unit 11 includes at least one input interface capable of receiving input of a captured image p1 from the image capturing device 20. The input unit 11 supports input of still images and moving images. The image capturing device 20 is, for example, a digital camera.

[0017] The storage unit 12 includes one or more memories. The memories may be, for example, semiconductor memories, magnetic memories, or optical memories, but are not limited to these. Each memory included in the storage unit 12 may function as, for example, a main storage device, an auxiliary storage device, or a cache memory. The storage unit 12 stores any information used in the operation of the detection device 10. For example, the storage unit 12 may store system programs, application programs, embedded software, and an AI engine that performs machine learning.

[0018] The output unit 13 includes at least one output interface capable of outputting image data to the display device 30. The output unit 13 is capable of outputting still images and moving images. The output unit 13 outputs a display image p3 generated by a control unit 14 (described later) to the display device 30. When a moving image is input to the input unit 11, the output unit 13 can output the display image p3 as a moving image by connecting still images generated for each frame.

[0019] The control unit 14 includes one or more processors, one or more programmable circuits, one or more dedicated circuits, or a combination thereof. The processor is, for example, a general-purpose processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), or a dedicated processor specialized for specific processing. However, the processor is not limited to these. The programmable circuit is, for example, an FPGA (Field-Programmable Gate Array). However, the programmable circuit is not limited to an FPGA. The dedicated circuit is, for example, an ASIC (Application Specific Integrated Circuit). However, the dedicated circuit is not limited to an ASIC. The control unit 14 performs information processing related to the operation of the detection device 10.

[0020] As shown in FIG. 1, the control unit 14 has a prediction function, a comparison function, and a detection function.

[0021] The prediction function is a function for predicting the pre-deterioration state of the concrete surface of a structure from a photographed image p1 of the structure. As shown in FIG. 2 , when the photographed image p1 is input to a classifier 14A constructed using machine learning, the control unit 14 extracts pixels of a concrete region Rc of the structure from the photographed image p1. Next, when the pixels of the extracted concrete region Rc are input to a generator 14B constructed using a generative adversarial network, the control unit 14 generates a surface image p2 that predicts the pre-deterioration state of the concrete surface of the structure. The pre-deterioration state is a state in which there is no concrete surface deterioration such as cracks, fissures, exposed rebar, water leakage, or free lime, and no concrete surface abnormalities such as wear on the concrete surface, spalling where parts have peeled off, or junk where aggregate is densely packed.

[0022] The comparison function is a function for identifying an image region Rd where there is a difference between the photographed image p1 and the surface image p2 obtained as a result of the previous prediction.

[0023] The deterioration function detects a predetermined deterioration d of the concrete surface in the identified image region Rd having a difference, and generates a display image p3 in which the image region Rd having a difference is divided into a first image region Ra1 where the predetermined deterioration d is detected and a second image region Ra2 where the predetermined deterioration is not detected on the captured image p1. The control unit 14 outputs the display image p3 to the output unit 13.

[0024] (Operation Flow of Detection Apparatus) Fig. 4 is a flowchart showing an operation example of the detection apparatus 10 according to an embodiment of the present disclosure. The operation of the detection apparatus 10 will be described in detail with reference to Fig. 4 .

[0025] S101 : The input unit 11 inputs a photographed image p1 of a structure photographed by the photographing device 20 .

[0026] The image capturing device 20 is, for example, a digital camera that captures still images and moving images, or a video camera that captures moving images, etc. The image capturing device 20 outputs a captured image p1 of a structure, which is a still image or a moving image, to the input unit 11.

[0027] When a captured image p1 of a structure is input as a video, the input unit 11 can create a still image for each frame. The frame interval may be set arbitrarily. The input unit 11 can input both video and still images. Therefore, the input unit 11 can select still images or video depending on the size of the structure to be inspected or the inspection method, thereby improving the efficiency of the inspection.

[0028] S102: The control unit 14 predicts the state of the concrete surface of the structure before deterioration from the photographed image p1 of the structure.

[0029] The control unit 14 constructs a determinator 14A that predicts pixels on the concrete surface by machine learning the photographed image p1 of the structure input from the input unit 11 and information that labels the pixels on the concrete surface to be predicted.

[0030] The control unit 14 extracts pixels of the concrete region Rc by inputting the captured image p1 of the structure into a classifier 14A constructed by machine learning. The pixels of the concrete region Rc include deterioration that has occurred on the concrete surface, such as cracks and peeling. Furthermore, if the number of pixels of the concrete region Rc is less than a predetermined standard, it may be determined that a concrete structure is not captured.

[0031] The control unit 14 has an extraction function for extracting pixels of the concrete region Rc, which eliminates the need to limit imaging to the concrete region Rc when imaging a structure, thereby improving imaging efficiency. Furthermore, when the concrete region Rc is small, the pixels of the concrete region Rc are not output to the generator 14B at the subsequent stage, thereby reducing processing time.

[0032] The control unit 14 uses a generative adversarial network to learn the pixels of the concrete surface in its pre-deterioration state, thereby constructing a generator 14B. A generative adversarial network is a machine learning model that generates new data by having two neural networks compete with each other. One of the two neural networks generates new data, and the other network identifies whether the data is genuine or fake. By inputting the extracted pixels of the concrete surface in its pre-deterioration state into the generator 14B constructed using the generative adversarial network, a surface image p2 that predicts the pre-deterioration state of the concrete surface of the structure is obtained.

[0033] The control unit 14 predicts the state of the concrete surface in the concrete region Rc before deterioration based on the extracted pixels, and generates a surface image p2 obtained as a result of the prediction. The pixels input to the generator 14B are pixels obtained by cutting out only the concrete region Rc from the photographed image p1 of the structure in the preceding determiner 14A so that only the extracted concrete region Rc is displayed. The control unit 14 generates the surface image p2 by inputting the pixels of the extracted concrete region Rc to the generator 14B. When the control unit 14 compares the surface image p2 with the photographed image p1 in S103, which will be described later, it can limit the image region Rd that has a difference to the concrete region Rc, thereby improving the accuracy of detecting deterioration that has occurred in the concrete region Rc.

[0034] Fig. 5 shows examples of a photographed image p1 and a surface image p2. The left image in Fig. 5 is an image in which only the concrete region Rc is cut out from the photographed image p1 of the structure. The right image in Fig. 5 is a surface image p2 obtained as a result of prediction.

[0035] S103: The control unit 14 identifies an image region Rd in the photographed image p1 that has a difference from the surface image p2 obtained as a result of the prediction.

[0036] 6 is a diagram illustrating division into rectangular regions. In the example shown in FIG. 6, the control unit 14 divides the captured image p1 and the surface image p2 into nine rectangular regions Rs of a predetermined size. The size of the rectangular regions Rs may be set arbitrarily.

[0037] The control unit 14 divides the captured image p1 into a plurality of rectangular regions Rs, and calculates an index (referred to as a first index PSNR) for evaluating the difference for each rectangular region Rs using the difference in luminance value between each pixel of the captured image p1 and the corresponding pixel of the surface image p2. The first index PSNR is calculated by the following formula (1). In formula (1), MAX ２ is the maximum value of the luminance. MSE is the mean square error of the luminance of corresponding pixels in the captured image p1 and the surface image p2.

[0038] The control unit 14 determines that a rectangular region Rs having a calculated first index PSNR value greater than a threshold value (referred to as a first threshold value α) is an image region Rd having a difference.

[0039] The control unit 14 further divides the captured image p1 into a plurality of rectangular regions Rs, and calculates an index (referred to as a second index SSIM) for evaluating the difference for each rectangular region Rs using information on a group of pixels within a certain range centered on each pixel of the captured image p1 and information on a group of pixels within a certain range centered on the corresponding pixel of the surface image p2. The second index SSIM is calculated by the following formula (2). In formula (2), x indicates the captured image p1, and y indicates the surface image p2. Furthermore, μ is the average pixel value of the surrounding pixels. The surrounding pixels are pixels surrounding the pixel of interest, within a predetermined range. The predetermined range may be set arbitrarily. б is the standard deviation of the pixel values ​​of the surrounding pixels. б x,y is the covariance of x and y. C1 and C2 are constants.

[0040] The control unit 14 determines that a rectangular region Rs in which the calculated value of the second index SSIM is greater than a threshold value (referred to as a second threshold value β) is an image region Rd in which a difference exists.

[0041] The selection of whether to use the first index PSNR or the second index SSIM is determined based on the resolution of the image or the extent to which degradation appears in the image.

[0042] S104: The control unit 14 determines whether or not an image region Rd having a difference has been identified. If an image region Rd having a difference has been identified, the process proceeds to S105. If an image region Rd having a difference has not been identified, the control unit 14 ends the information processing.

[0043] 7 is a diagram illustrating the result of identifying the image regions Rd having differences. In the example shown in Fig. 7, the control unit 14 identifies six rectangular regions Rs as the image regions Rd having differences.

[0044] S105: The control unit 14 detects a predetermined deterioration d of the concrete surface in the identified image region Rd having a difference. In both cases where the predetermined deterioration d is detected and where the predetermined deterioration d is not detected, the process proceeds to S105.

[0045] The control unit 14 includes a detector 14C constructed by machine learning to detect pixels that constitute a predetermined degradation d. The control unit 14 detects the presence or absence of the predetermined degradation d by inputting an image of an image region Rd having a difference to the detector 14C. When the predetermined degradation d is detected, the control unit 14 identifies a first image region Ra1 in which the predetermined degradation d is detected and determines the type of degradation.

[0046] The predetermined deterioration d is assumed to be, but is not limited to, cracks, fissures, exposed rebars, water leakage, and free lime deterioration. The type of deterioration is cracks, fissures, exposed rebars, water leakage, and free lime deterioration.

[0047] The control unit 14 inputs the image and label information assigned to the pixels of the deterioration to be detected, and constructs a detector 14C using machine learning to detect pixels where deterioration (cracks, fissures, exposed rebar, water leakage, or free lime) has occurred. Note that the label information used to detect the rectangular region Rs is the label information of the deterioration depicted in the rectangular region Rs. A detector 14C may be generated for each type of deterioration. The control unit 14 detects a predetermined deterioration d using the constructed detector 14C.

[0048] S106: The control unit 14 generates a display image p3 that distinguishes, on the captured image p1, a first image area Ra1 in which a specified degradation d is detected and a second image area Ra2 in which the specified degradation d is not detected, of the image area Rd with a difference.

[0049] The second image region Ra2 is a region that may contain abnormalities such as worn concrete surface, spalling where parts have peeled off, or junk where aggregate is densely packed. Figure 8 is a diagram showing an example of a display image p3. In the example shown in Figure 8, three rectangular regions Rs are determined to be the first image region Ra1 where exposed rebar corresponding to a predetermined deterioration d has been detected, and another three rectangular regions Rs are determined to be the second image region Ra2. This makes it possible to identify the second image region Ra2 where the predetermined deterioration d is not detected but where minor abnormalities on the concrete surface of the structure are detected.

[0050] S107: The output unit 13 outputs the display image p3 generated by the control unit 14 to the display device 30.

[0051] When a moving image is input to the input unit 11, the output unit 13 may output the display image p3 as a moving image by connecting still images created for each frame.

[0052] As described above, the detection device 10 of this embodiment is equipped with a control unit 14 that predicts the state of the concrete surface of a structure before deterioration from a photographed image of the structure, identifies image areas in the photographed image that differ from the surface image obtained as a result of the prediction, detects specified deterioration of the concrete surface in the identified image areas that differ, and generates a display image in which a first image area in which the specified deterioration was detected and a second image area in which the specified deterioration was not detected are distinguished on the photographed image, and an output unit 13 that outputs the display image generated by the control unit 14 to a display device 30.

[0053] With this configuration, an image region Rd is identified that contains a difference between a captured image p1 of the structure and a surface image p2 of the structure's concrete surface before deterioration, obtained as a result of prediction. The identified image region Rd that contains a difference is then distinguished into a first image region Ra1 in which a predetermined degree of deterioration d is detected and a second image region Ra2 in which the predetermined degree of deterioration d is not detected. By closely examining the distinguished second image region Ra2, it is possible to detect small abnormalities in the concrete surface, such as wear on the concrete surface, spalling where parts have peeled off, and junk where aggregate is densely packed.

[0054] Although the present disclosure has been described based on the drawings and examples, it should be noted that those skilled in the art may make various modifications and alterations based on the present disclosure. Therefore, it should be noted that these modifications and alterations are included in the scope of the present disclosure. For example, the functions included in each component or step can be rearranged so as not to be logically inconsistent, and multiple components or steps can be combined or divided into one.

[0055] For example, in the above-described embodiment, the configuration and operation of the detection device 10 may be distributed among a plurality of computers that can communicate with each other.

[0056] Also, an embodiment is possible in which, for example, a general-purpose computer functions as the detection device 10 according to the above-described embodiment. FIG. 9 is a block diagram showing a schematic configuration example of a computer functioning as the detection device 10. Specifically, a program describing the processing content for realizing each function of the detection device 10 according to the above-described embodiment is stored in the memory of the general-purpose computer, and the program is read and executed by a processor. Therefore, the present disclosure can also be realized as a program executable by a processor, or a non-transitory computer-readable medium storing the program.

[0057] The following additional notes are provided regarding the above-described embodiments.

[0058] (Supplementary Item 1) A detection device comprising: a control unit that predicts the state of a concrete surface of a structure before deterioration from a photographed image of the structure, identifies an image area of ​​the photographed image that differs from a surface image obtained as a result of the prediction, detects predetermined deterioration of the concrete surface in the identified image area with the difference, and generates a display image in which a first image area of ​​the image area with the difference where the predetermined deterioration is detected and a second image area where the predetermined deterioration is not detected are distinguished on the photographed image, and an output unit that outputs the display image generated by the control unit to a display device. (Supplementary Item 2) A detection device according to Supplementary Item 1, wherein the control unit extracts pixels of the concrete area by inputting the photographed image to a classifier constructed by machine learning, and obtains the surface image by inputting the extracted pixels to a generator constructed by learning using a generative adversarial network. (Supplementary Item 3) The detection device according to Supplementary Item 1 or 2, wherein the control unit divides the captured image into a plurality of rectangular regions, and for each rectangular region, calculates an index for evaluating the difference using the difference in luminance values ​​between each pixel of the captured image and a corresponding pixel of the surface image, and determines a rectangular region in which the calculated index value is greater than a threshold to be the image region in which the difference exists. (Supplementary Item 4) The detection device according to Supplementary Item 1 or 2, wherein the control unit divides the captured image into a plurality of rectangular regions, and for each rectangular region, calculates an index for evaluating the difference using information on a group of pixels in a certain range centered on each pixel of the captured image and information on a group of pixels in a certain range centered on the corresponding pixel of the surface image, and determines a rectangular region in which the calculated index value is greater than a threshold to be the image region in which the difference exists.

[0059] REFERENCE SIGNS LIST 1 System 10 Detection device 11 Input unit 12 Storage unit 13 Output unit 14 Control unit 20 Imaging device 30 Display device 13A Determinator 13B Generator 14C Detector

Claims

1. A detection device comprising: a control unit that predicts the state of the concrete surface of a structure before deterioration from a photographed image of the structure, identifies an image area of ​​the photographed image that is different from the surface image obtained as a result of the prediction, detects specified deterioration of the concrete surface in the identified image area with the difference, and generates a display image on the photographed image that distinguishes between a first image area where the specified deterioration is detected and a second image area where the specified deterioration is not detected, among the image areas with the difference; and an output unit that outputs the display image generated by the control unit to a display device.

2. A detection device as described in claim 1, wherein the control unit extracts pixels of concrete areas by inputting the captured image into a classifier constructed by machine learning, and obtains the surface image by inputting the extracted pixels into a generator constructed by learning using a generative adversarial network.

3. A detection device according to claim 1, wherein the control unit divides the captured image into a plurality of rectangular regions, calculates an index for evaluating the difference for each rectangular region using the difference in brightness between each pixel of the captured image and the corresponding pixel of the surface image, and determines that a rectangular region in which the calculated index value is greater than a threshold value is an image region in which the difference exists.

4. A detection device according to claim 1, wherein the control unit divides the captured image into a plurality of rectangular regions, and for each rectangular region, calculates an index for evaluating the difference using information on a group of pixels in a certain range centered on each pixel of the captured image and information on a group of pixels in a certain range centered on the corresponding pixel of the surface image, and determines that a rectangular region in which the calculated index value is greater than a threshold value is an image region in which the difference exists.

Images