Image processing device, image processing method and program

Abstract

An image processing device (10) according to the present disclosure comprises: a depth estimating unit (12) for estimating a depth, which is a distance in a depth direction in an image, for each pixel constituting the image; an inspection region setting unit (13) for setting, as a depth threshold, a depth of a region of a target object for which a ratio of a size of the target object to a size of a degradation to be inspected in the image is equal to or greater than a prescribed value, and, on the basis of the depth for each pixel, estimated by the depth estimating unit (12), setting a region closer than the depth threshold as an inspection region; and a result output unit (14) for outputting a setting result of the inspection region obtained by the inspection region setting unit (13).

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] Currently, infrastructure inspections are mainly performed visually by inspectors. As an alternative to visual inspection, deterioration detection technology using image recognition is being widely studied. For example, Non-Patent Documents 1 and 2 describe a method for detecting cracks in concrete or asphalt. The methods described in Non-Patent Documents 1 and 2 use deep learning techniques to detect minute cracks in acquired images with high accuracy. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Mohamed Alipour, Devin K Harris, and Gregory R miller SMS et al. “Robust Pixel-Level Crack Detection Using Convolutional Neural Networks”, Journal of Computing in Civil Engineering, 33(6), 04019040, 2019. [Non-patent document 2] K. Zenko, A. Igo, Y. Minamimenra, K. Kuroki, and K. Okubo, "Evaluation of crack rate using deep learning from pavement images captured by an in-vehicle camera," Journal of the Japan Society of Civil Engineers (E1), Vol. 73, No. 3, pp. I_97-I_105, 2017. Summary of the Invention [Problem to be solved by the invention]

[0004] The methods described in Non-Patent Documents 1 and 2 are based on the premise that only concrete surfaces are captured in the acquired images, and therefore have limitations on the imaging, such as the requirement that only concrete surfaces be included within the camera's angle of view. To efficiently inspect infrastructure facilities on-site using images, it is necessary to use images of various facilities captured simultaneously using equipment such as an MMS (Mobile Mapping System), rather than images captured under the limitations described above. However, when attempting to capture images of various infrastructure facilities simultaneously using an MMS or similar device, it is difficult to capture the size of the infrastructure facilities being inspected uniformly in the captured images, and the size of the infrastructure facilities in the images may be so large that it is impossible to determine whether or not they have deteriorated.

[0005] Consider, for example, the case of capturing infrastructure facilities such as guardrails or signs using an MMS. In this case, it is difficult to maintain a constant shooting distance from the vehicle due to factors such as the lane of travel of the camera-equipped vehicle, the road alignment, and the location of the infrastructure facilities. It is also difficult to consistently capture infrastructure facilities at a consistent size or composition in the captured image. As a result, infrastructure facilities located far away may appear within the field of view of the captured image. Areas where distant infrastructure facilities are captured may not be detectable, making them undesirable for inspection. Therefore, a method is needed to appropriately determine inspection areas for inspecting infrastructure facilities and other objects for deterioration in images captured.

[0006] The purpose of the present disclosure, made in consideration of the above-mentioned problems, is to provide an image processing device, an image processing method, and a program that can appropriately determine an inspection area for inspecting whether or not an object, such as infrastructure equipment, has deteriorated in an image of the object. [Means for solving the problem]

[0007] In order to solve the above problem, the image processing device of the present disclosure is an image processing device that determines an inspection area in an image of an object photographed to check for the presence or absence of deterioration of the object, and is equipped with a depth estimation unit that estimates the depth, which is the distance in the depth direction in the image, for each pixel that constitutes the image, an inspection area determination unit that determines the depth of the area of ​​the object in the image where the ratio of the size of the object to the size of the deterioration to be inspected is equal to or greater than a predetermined value as a depth threshold, and determines an area closer to the depth threshold as the inspection area based on the depth of each pixel estimated by the depth estimation unit, and a result output unit that outputs the inspection area determination result by the inspection area determination unit.

[0008] In addition, in order to solve the above problem, the image processing method disclosed herein is an image processing method by an image processing device that determines an inspection area in an image of an object photographed to check for the presence or absence of deterioration of the object, and includes the steps of: estimating the depth, which is the distance in the depth direction in the image, for each pixel that constitutes the image; determining the depth of the area of ​​the object in the image where the ratio of the size of the object to the size of the deterioration to be inspected is equal to or greater than a predetermined value as a depth threshold; determining an area closer to the depth threshold as the inspection area based on the estimated depth for each pixel; and outputting the inspection area determination result.

[0009] In order to solve the above problem, a program according to the present disclosure causes a computer to operate as the image processing device described above. [Effects of the Invention]

[0010] According to the image processing device, image processing method, and program of the present disclosure, it is possible to appropriately determine an inspection area in an image of an object, in which the presence or absence of deterioration of the object is inspected. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an image processing device according to an embodiment of the present disclosure. [Figure 2] 2 is a diagram for explaining depth estimation by a depth estimation unit shown in FIG. 1. FIG. [Figure 3] 2 is a flowchart showing an example of the operation of the image processing device shown in FIG. [Figure 4] 2 is a flowchart showing an example of the operation of a determination unit shown in FIG. [Figure 5] FIG. 10 is a diagram for explaining the number of pixels of an object required for inspection. [Figure 6A] This is an image of a guardrail as an object. [Figure 6B] This is an annotation image in which information on the pixel area of ​​the guardrail is added to the image shown in FIG. 6A. [Figure 6C] FIG. 6B is a diagram showing the depth estimation results of the pixel region of the guardrail in the image shown in FIG. 6A. [Figure 7] FIG. 10 is a diagram showing an example of the distribution of GR width and depth. [Figure 8] FIG. 8 is a diagram showing an example of an index (Accuracy) calculated based on the distribution shown in FIG. 7. [Figure 9] 1. FIG. 4 is a diagram for explaining another example of a method for determining a depth threshold α by the threshold determination unit shown in FIG. [Figure 10] 1. FIG. 6 is a diagram illustrating yet another example of a method for determining a depth threshold α by the threshold determination unit shown in FIG. [Figure 11] 2 is a diagram illustrating an example of the configuration of a threshold value determining unit illustrated in FIG. 1; [Figure 12] FIG. 2 is a diagram illustrating an example of a hardware configuration of the image processing device illustrated in FIG. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0013] 1 is a diagram illustrating an example configuration of an image processing device 10 according to an embodiment of the present disclosure. The image processing device 10 according to this embodiment determines an inspection area for inspecting an object, such as infrastructure equipment, for deterioration in an image captured by a monocular camera.

[0014] As shown in FIG. 1, an image processing device 10 according to this embodiment includes an image input unit 11, a depth estimation unit 12, an inspection region determination unit 13, and a result output unit .

[0015] An image of an object such as infrastructure equipment is input to the image input unit 11. The image input unit 11 outputs the input image to the depth estimation unit 12.

[0016] The depth estimation unit 12 estimates the depth, which is the distance in the depth direction, of each pixel constituting the image output from the image input unit 11. The depth estimation unit 12 estimates the depth of each pixel using, for example, a depth estimation model (e.g., Midas) based on a convolutional neural network (CNN), which is a type of deep learning method.

[0017] Fig. 2 is a diagram showing an example of a result of depth estimation by the depth estimation unit 12. Fig. 2 shows an example of an output image that the depth estimation unit 12 outputs as a result of depth estimation when an image of a guardrail is used as an input image to the depth estimation unit 12.

[0018] The depth estimation unit 12 inputs an input image into a depth estimation model, and outputs an output image in which the relative distance in the depth direction for each pixel is indicated by a numerical value (0 (black) to 255 (white)), as shown in FIG. 2. By generating such an image, it becomes possible to estimate how far an object in the image is. Note that FIG. 2 shows an example in which the smaller the distance in the depth direction, the larger the value indicating the depth, and the larger the distance in the depth direction, the smaller the value indicating the depth, but this is not limiting.

[0019] Referring back to FIG. 1, the depth estimation unit 12 outputs the estimation result to the inspection area determination unit 13.

[0020] The inspection area determination unit 13 determines the depth of an object area in the image where the ratio of the size of the object to the size of the degradation to be inspected is equal to or greater than a predetermined value as a depth threshold α to be used for determining the inspection area. Then, based on the depth of each pixel estimated by the depth estimation unit 12, the inspection area determination unit 13 determines an area closer to the depth threshold α (an area closer to the depth threshold α) as the inspection area. As shown in FIG. 1, the inspection area determination unit 13 includes a threshold determination unit 131 and a determination unit 132.

[0021] The threshold determination unit 131 determines a depth threshold α. In this embodiment, a value between 0 and 255 is set as the depth for each pixel, so the depth threshold α is 0 or more and 255 or less (0≦α≦255). However, this is not limited to this, and the depth threshold α can be set appropriately depending on how the depth for each pixel is expressed. Details of how the threshold determination unit 131 determines the depth threshold α will be described later.

[0022] The determination unit 132 determines, based on the depth of each pixel estimated by the depth estimation unit 12, an area closer to the depth threshold α determined by the threshold determination unit 131 as the inspection area.

[0023] The result output unit 14 outputs the result of the determination of the inspection area by the inspection area determination unit 13. The result output unit 14 may, for example, extract and output an image of the area determined to be the inspection area by the inspection area determination unit 13 from the image input to the image input unit 11. Furthermore, the result output unit 14 may, for example, superimpose an image showing the inspection area determined by the inspection area determination unit 13 on the image input to the image input unit 11 and output the image.

[0024] Next, the operation of the image processing device 10 according to this embodiment will be described.

[0025] FIG. 3 is a flowchart showing an example of the operation of the image processing device 10 according to this embodiment, and is a diagram for explaining an image processing method executed by the image processing device 10 according to this embodiment.

[0026] The depth estimation unit 12 estimates the depth for each pixel constituting the image of the object that is input via the image input unit 11 (step S11). As described above, the depth estimation unit 12 estimates the depth for each pixel using a depth estimation model such as Midas.

[0027] The inspection area determination unit 13 (threshold determination unit 131) determines the depth of an area where the ratio of the size of the object to the size of the deterioration to be inspected is equal to or greater than a predetermined value as a depth threshold α. Then, the inspection area determination unit 13 (determination unit 132) determines an area closer to the depth threshold α as the inspection area based on the depth of each pixel estimated by the depth estimation unit 12 (step S12).

[0028] FIG. 4 is a diagram for explaining the operation of determining an inspection area by the inspection area determination unit 13 (determination unit 132).

[0029] The determining unit 132 determines the depth p of the pixel i, j estimated by the depth estimating unit 12. i,j is acquired (step S121).

[0030] Next, the determination unit 132 determines the depth p i,j is greater than or equal to the depth threshold α (p i,j ≧α) (step S122).

[0031] p i,j If it is determined that the depth threshold α is satisfied (the pixel i, j is in a region closer than the depth threshold α) (step S122: Yes), the determination unit 132 outputs the determination result R i,j =1 (step S123).

[0032] p i,jIf it is determined that ≧α is not satisfied (step S122: No), the determination unit 132 determines the determination result R i,j =0 (step S124).

[0033] When the above-described process is completed for all pixels, the determination unit 132 outputs the determination result R i,j 4, the inspection area determination unit 13 (determination unit 132) does not determine an area farther than the depth threshold value α as the inspection area, but can determine only an area closer than the depth threshold value α as the inspection area.

[0034] Referring again to FIG. 3, the result output unit 14 outputs the result of the determination of the inspection area by the inspection area determination unit 13 (step S13).

[0035] Next, a method for determining the depth threshold α by the threshold determination unit 131 will be described.

[0036] In this embodiment, the number of pixels N of an object on an image required for inspecting the object is set in advance, and the threshold determination unit 131 determines the depth threshold α based on the number of pixels N. Specifically, the threshold determination unit 131 determines the depth threshold α based on the size of the object in the image and the distribution of the depth of the area of ​​the object having that size (hereinafter referred to as the "size-depth distribution of the object").

[0037] FIG. 5 is a diagram for explaining a method for determining the number N of pixels of an object required for inspection.

[0038] As shown in Figure 5, the size of the object to be inspected is defined as x. x is the size in the direction parallel to the short side of the object to be inspected. Also, as shown in Figure 5, the size of the minimum deterioration that we want to detect through inspection is defined as y. The size of the deterioration y is the size in the same direction as the size x of the object to be inspected.

[0039] If the number of pixels N of an object is x / y or greater (N≧x / y), then if there is a degradation of size y, that degradation will span more than one pixel, making it possible to detect the degradation. For example, suppose the size of the object x is 35 cm and the size of the degradation y is 7 cm. In this case, since 35 / 7=5, if the number of pixels N of the object is 5 or more pixels in the same direction as the direction in which the size x of the object was determined, the degradation to be inspected will span more than one pixel, making it possible to detect the degradation. On the other hand, if the number of pixels N of the object is less than 5 pixels, the degradation to be inspected will have less information than one pixel, making it difficult to detect the degradation. Thus, if the number of pixels of the object (the size of the object on the image) is N or greater, the desired degradation can be detected by inspection. Therefore, it is clear that the inspection area should be determined to be the area on the image where the number of pixels of the object is N or greater.

[0040] Next, we will explain how to extract an area where N≧x / y (an area of ​​an object where the ratio of the size x of the object to the size y of the deterioration to be inspected is equal to or greater than a predetermined value). In the following, we will explain the example where the object is a guardrail.

[0041] FIG. 6A is an image of a guardrail captured by an in-vehicle camera. FIG. 6B is an annotation image of the image shown in FIG. 6A, in which "1" is assigned to the pixel region of the guardrail and "0" is assigned to pixels in other regions. FIG. 6C is an image showing the depth estimation result of only the pixel region of the guardrail using a depth estimation model. In FIGS. 6A to 6C, the horizontal direction on the paper surface is referred to as the x direction, and the vertical direction on the paper surface is referred to as the y direction. Note that the annotation image shown in FIG. 6B can be created, for example, by performing object recognition processing on an image input via image input unit 11.

[0042] The guardrail is composed of a beam with a wavy cross section and a support post that supports the beam. The size of the guardrail is defined by the number of pixels in the y direction in the pixel area of ​​the guardrail in the image. The threshold determination unit 131 defines the coordinates of the upper left pixel in the annotation image shown in FIG. 6B as (1, 1), counts the number of pixels that have a value of 1 input in the y direction for each x coordinate, and determines the counted number as the guardrail width at that x coordinate (hereinafter referred to as the "GR width"). For example, in FIG. 6B, the dashed line portion has an x ​​coordinate of 100 and a GR width of 48. As can be seen from FIG. 6A, the GR width decreases as the x coordinate increases.

[0043] Next, the threshold determination unit 131 calculates the depth of the region of the object having the GR width determined based on the annotation image. Specifically, the threshold determination unit 131 defines the coordinates of the upper left pixel in the image shown in FIG. 6C as (1, 1) and sums the depth values ​​of each pixel in the pixel region of the guardrail in the y direction. The threshold determination unit 131 then divides the sum of the depth values ​​for each x coordinate by the GR width at that x coordinate to calculate the depth of the region of the object having the GR width at that x coordinate. For example, in FIG. 6C, the x coordinate of the dashed line portion is 100, and the depth of the region of the object having the GR width is 183.

[0044] Figure 7 is a plot of the relationship between the GR width and the depth of the area of ​​the object (guardrail) having that GR width (size-depth distribution of the object), obtained from 4,400 sample images. For example, if the area of ​​the guardrail that can be inspected is defined as an area where the GR width is N=20 pixels or more, by setting a depth threshold α, the size-depth distribution of the object can be classified into four areas, as shown in Figure 7.

[0045] The upper right region is a region where the GR width is 20 pixels or more and the depth is the depth threshold α or more (the region where the ratio (x / y) of the object size x to the degradation size y is a predetermined value (N=20) or more and the depth is the depth threshold α or more). This region is defined as TP. TP is a region where an image region with a GR width of 20 pixels or more is correctly extracted as a valid range for inspection.

[0046] The area in the lower right is an area where the GR width is less than 20 pixels and the depth is greater than or equal to the depth threshold α (an area where the ratio (x / y) of the object size x to the degradation size y is less than a predetermined value (N=20) and the depth is greater than or equal to the depth threshold α). This area is defined as FP. FP is an area where an image area with a GR width of less than 20 pixels is mistakenly extracted as a valid area for inspection.

[0047] The upper left area is an area where the GR width is 20 pixels or more and the depth is less than the depth threshold α (an area where the ratio (x / y) of the size x of the object to the size y of the deterioration to be inspected is greater than a predetermined value (N=20) and the depth is less than the depth threshold α). This area is defined as FN. FN is an area where image areas with a GR width of 20 pixels or more are mistakenly excluded as areas that are not valid for inspection.

[0048] The area on the bottom left is an area where the GR width is less than 20 pixels and the depth is less than the depth threshold α (an area where the ratio (x / y) of the object size x to the degradation size y is less than a predetermined value (N=20) and the depth is less than the depth threshold α). This area is defined as TN. TN is an area where image areas with a GR width less than 20 pixels are correctly excluded as areas that are not valid for inspection.

[0049] The threshold determination unit 131 calculates the index (Accuracy) based on the above classification, for example, using formula (1). Note that TP, FP, FN, and TN in formula (1) are the numbers of data classified into the regions TP, FP, FN, and TN, respectively. Accuracy=(TP+TN) / (TP+FP+FN+TN) Formula (1)

[0050] The threshold determination unit 131 determines the depth threshold α based on the index (Accuracy) calculated according to the above-described formula (1). Specifically, the threshold determination unit 131 determines the depth at which the index (Accuracy) calculated based on formula (1) is the largest as the depth threshold α.

[0051] Fig. 8 is a diagram showing an example of the index (Accuracy) calculated according to formula (1) based on the distribution shown in Fig. 7. As shown in Fig. 8, when the depth threshold α=73, the index (Accuracy) is highest (the classification accuracy is highest). Therefore, the threshold determination unit 131 determines the depth threshold α to be 73.

[0052] The index used to determine the depth threshold α is not limited to the above-mentioned index (Accuracy). The threshold determination unit 131 may determine the depth threshold α using, for example, an index (Recall) calculated according to the following formula (2): Recall=TP / (TP+FN) Formula (2)

[0053] When the index (Recall) calculated according to Equation (2) is used, even areas that may be suitable for the inspection area may be excluded from the decision area. In other words, by using the index (Recall), it is possible to determine the depth threshold α such that inappropriate areas are less likely to be included in the inspection area.

[0054] Furthermore, the threshold value determination unit 131 may determine the depth threshold value α using an index (Specificity) calculated according to the following equation (3), for example. Specificity=TN / (FP+TN) Formula (3)

[0055] When the index (Specificity) calculated according to Equation (3) is used, even areas that may not be included in the inspection area may be extracted as inspection areas. In other words, by using the index (Specificity), it is possible to determine the depth threshold α so that a wider area is included in the inspection area.

[0056] The threshold determination unit 131 creates a size-depth distribution of an object as shown in Figure 7 based on the GR width and the depth of the area having that GR width calculated for each x-coordinate of the input image, and determines the depth threshold α using the above-mentioned indicators.

[0057] In this embodiment, the object is a guardrail, but the object is not limited to a guardrail. The threshold value determination unit 131 sets TP, FP, FN, and TN for any object based on the size-depth distribution of the object, and determines the depth threshold value α using the indexes described above.

[0058] In the above example, the threshold determination unit 131 sets TP, FP, FN, and TN based on the size-depth distribution of the object and determines the depth threshold α using the above-mentioned index, but the present invention is not limited to this. For example, the threshold determination unit 131 may set an approximation line (an approximation line or an approximation curve) for the size-depth distribution of the object to determine the depth threshold α.

[0059] Specifically, the threshold determination unit 131 sets an approximation line (an approximation straight line in FIG. 9) for the size-depth distribution of the object, as shown in FIG. 9. Then, the threshold determination unit 131 determines the depth corresponding to the size of the object on the approximation line where the ratio (x / y) of the object size x to the degradation size y is equal to or greater than a predetermined value (N=20) as the depth threshold α. In the example shown in FIG. 9, if N=20, then 47, which is the depth on the approximation line corresponding to a GR width of 20 pixels, is determined as the depth threshold α.

[0060] Furthermore, the threshold determination unit 131 may set a plurality of approximation lines for the size-depth distribution of an object. For example, as shown in Fig. 10, the threshold determination unit 131 may set approximation lines for size-depth distributions of an object with different slopes. In the example shown in Fig. 10, if N = 20 in image 1, the depth corresponding to a GR width of 20 pixels is 80. If N = 20 in image 2, the depth corresponding to a GR width of 20 pixels is 70.

[0061] In this case, the threshold value determining unit 131 includes a classification model 1311 as shown in FIG.

[0062] The classification model 1311 is a model that classifies an input image according to the size-depth distribution of an object in the image. For example, the classification model 1311 classifies the input image according to the gradient of the size-depth distribution of an object in the image. The classification model 1311 is constructed using a statistical discrimination model such as deep learning.

[0063] The threshold determination unit 131 sets an approximation line for the size-depth distribution of an object in an image for each classification of the input image by the classification model 1311. Then, the threshold determination unit 131 determines the depth threshold α as the depth of an object region where the ratio (x / y) of the object size x to the degradation size y is equal to or greater than a predetermined value on the approximation line set for the image classified by the classification model 1311. For example, if N=20 and the input image is classified as image 1 by the classification model 1311, the threshold determination unit 131 sets the depth threshold α to 80, which is the depth corresponding to a region with a GR width of 20 pixels on the approximation line set for image 1. Also, for example, if N=20 and the input image is classified as image 2 by the classification model 1311, the threshold determination unit 131 sets the depth threshold α to 70, which is the depth of a region with a GR width of 20 pixels on the approximation line set for image 2. This process enables the inspection region to be determined with higher accuracy than when a single approximation formula is used.

[0064] Next, the hardware configuration of the image processing device 10 according to this embodiment will be described.

[0065] Fig. 12 is a diagram showing an example of the hardware configuration of an image processing device 10 according to this embodiment. Fig. 12 shows an example of the hardware configuration of the image processing device 10 when the image processing device 10 is configured by a computer capable of executing program instructions. Here, the computer may be a general-purpose computer, a dedicated computer, a workstation, a PC (Personal Computer), an electronic notepad, etc. The program instructions may be program code, code segments, etc. for performing necessary tasks.

[0066] 12, the image processing device 10 includes a processor 21, a ROM (Read Only Memory) 22, a RAM (Random Access Memory) 23, a storage 24, an input unit 25, a display unit 26, and a communication interface (I / F) 27. Each component is connected to each other so as to be able to communicate with each other via a bus 29. The processor 21 is specifically a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an SoC (System on a Chip), or the like, and may be configured by a plurality of processors of the same type or different types.

[0067] The processor 21 is a control unit that controls each component and performs various arithmetic processing. That is, the processor 21 reads a program from the ROM 22 or the storage 24 and executes the program using the RAM 23 as a work area. The processor 21 controls each component and performs various arithmetic processing in accordance with the program stored in the ROM 22 or the storage 24. In this embodiment, the ROM 22 or the storage 24 stores a program for operating a computer as the image processing device 10 according to the present disclosure. The program is read and executed by the processor 21, thereby realizing each component of the image processing device 10, namely, the image input unit 11, the depth estimation unit 12, the inspection area determination unit 13, and the result output unit 14.

[0068] The program may be provided in a form stored on a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), a USB (Universal Serial Bus) memory, etc. The program may also be provided in a form downloaded from an external device via a network.

[0069] The ROM 22 stores various programs and various data. The RAM 23 temporarily stores programs or data as a working area. The storage 24 is configured with an HDD (Hard Disk Drive) or SSD (Solid State Drive) and stores various programs including an operating system and various data. The ROM 22 or storage 24 stores, for example, frames output as extracted frames.

[0070] The input unit 25 includes a pointing device such as a mouse and a keyboard, and is used to input various types of information.

[0071] The display unit 26 is, for example, a liquid crystal display, and displays various information. The display unit 26 may function as the input unit 25 by adopting a touch panel system.

[0072] The communication interface 27 is an interface for communicating with other devices (for example, a camera that has photographed an object), and is, for example, an interface for a LAN.

[0073] A computer can be suitably used to function as each unit of the image processing device 10 described above. Such a computer can be realized by storing a program describing the processing content for realizing the function of each unit of the image processing device 10 in a memory unit of the computer, and having the processor of the computer read and execute the program. In other words, the program can cause the computer to function as the image processing device 10 described above. The program can also be stored in a non-transitory storage medium. The program can also be provided via a network.

[0074] As described above, the image processing device 10 according to this embodiment includes a depth estimation unit 12, an inspection area determination unit 13, and a result output unit 14 that outputs the determination result of the inspection area determination unit 13. The depth estimation unit 12 estimates the depth for each pixel constituting an image. The inspection area determination unit 13 determines the depth of an object area in the image where the ratio of the size x of the object to the size y of the deterioration to be inspected is equal to or greater than a predetermined value as a depth threshold α. Then, based on the depth for each pixel estimated by the depth estimation unit 12, the inspection area determination unit 13 determines an area closer to the depth threshold α as the inspection area.

[0075] The depth of the region of the object where the ratio of the size x of the object to the size y of the deterioration is equal to or greater than a predetermined value is set as the depth threshold α, and the region closer to the depth threshold α is determined as the inspection region, so that the size of the deterioration in the determined region is equal to or greater than the size y of the deterioration to be detected. Therefore, in an image of an object such as infrastructure equipment, it is possible to determine as the inspection region an area where the presence or absence of deterioration of the object can be inspected.

[0076] The following additional notes are provided regarding the above-described embodiments.

[0077] [Additional note 1] An image processing device that determines an inspection area in an image of an object to inspect for the presence or absence of deterioration of the object, Memory and a control unit connected to the memory; Equipped with The control unit Estimating a depth, which is a distance in a depth direction in the image, for each pixel constituting the image; A depth threshold is determined as a depth of a region of the object in the image where a ratio of a size of the object to a size of the deterioration to be inspected is equal to or greater than a predetermined value, and a region closer to the depth threshold is determined as the inspection region based on the estimated depth of each pixel; an image processing device that outputs the result of determining the inspection area;

[0078] [Additional note 2] In the image processing device according to claim 1, The control unit determines the depth threshold based on a distribution of the size of the object in the image and the depth of a region of the object having that size.

[0079] [Additional note 3] In the image processing device according to Supplementary Item 2, In the distribution, the number of data in the region where the ratio is equal to or greater than the predetermined value and the depth is equal to or greater than the depth threshold is denoted as TP, the number of data in the region where the ratio is less than the predetermined value and the depth is equal to or greater than the depth threshold is denoted as FP, the number of data in the region where the ratio is equal to or greater than the predetermined value and the depth is less than the depth threshold is denoted as FN, and the number of data in the region where the ratio is less than the predetermined value and the depth is less than the depth threshold is denoted as TN. The control unit determines the depth threshold based on an index obtained from (TP+TN) / (TP+FP+FN+TN), TP / (TP+FN), or TN / (FP+TN).

[0080] [Additional note 4] In the image processing device according to Supplementary Item 2, The control unit sets an approximation line for the distribution and determines the depth threshold as the depth corresponding to the size of the object on the approximation line at which the ratio is equal to or greater than the predetermined value.

[0081] [Additional note 5] In the image processing device according to Supplementary Item 2, The control unit is provided with a classification model that classifies an input image according to the distribution, sets an approximation line for the distribution for each classification of the input image by the classification model, and determines the depth threshold to be the depth corresponding to the size of the object at which the ratio is greater than or equal to the predetermined value on the approximation line set for the image classified by the classification model.

[0082] [Additional note 6] 1. An image processing method for an image processing device that determines an inspection area in an image of an object, in which the presence or absence of deterioration of the object is inspected, the method comprising: Estimating a depth, which is a distance in a depth direction in the image, for each pixel constituting the image; A depth threshold is determined as a depth of a region of the object in the image where a ratio of a size of the object to a size of the deterioration to be inspected is equal to or greater than a predetermined value, and an area closer to the depth threshold is determined as the inspection area based on the estimated depth of each pixel; and outputting the result of determining the inspection area.

[0083] [Additional note 7] A non-transitory storage medium storing a program executable by a computer, the program causing the computer to operate as the image processing device described in any one of appendix 1 to 5.

[0084] Although the above-described embodiments have been described as typical examples, it will be apparent to those skilled in the art that many modifications and substitutions can be made within the spirit and scope of the present disclosure. Therefore, the present invention should not be interpreted as being limited by the above-described embodiments, and various modifications and alterations are possible without departing from the scope of the claims. For example, multiple building blocks shown in the block diagrams of the embodiments can be combined into one, or one building block can be divided. [Explanation of symbols]

[0085] 10 Image processing device 11 Image input unit 12 Depth estimation section 13 Inspection area determination unit 14 Result output section 131 Threshold determination unit 132 Judgment section 1311 Classification Model 21 processors 22 ROM 23 RAM 24 Storage 25 Input section 26 Display section 27 Communication I / F 29 Bus

Claims

1. An image processing device that determines an inspection area in an image of an object to inspect for the presence or absence of deterioration of the object, a depth estimation unit that estimates a depth, which is a distance in a depth direction in the image, for each pixel that constitutes the image; an inspection area determination unit that determines a depth of an area of ​​the object in the image where the ratio of the size of the object to the size of the deterioration to be inspected is equal to or greater than a predetermined value as a depth threshold, and determines an area closer to the depth threshold as the inspection area based on the depth of each pixel estimated by the depth estimation unit; a result output unit that outputs a result of the inspection area determination by the inspection area determination unit.

2. 2. The image processing device according to claim 1, The inspection area determination unit determines the depth threshold based on a distribution of the size of the object in the image and the depth of the area of ​​the object having that size.

3. 3. The image processing device according to claim 2, In the distribution, a region where the ratio is equal to or greater than the predetermined value and the depth is equal to or greater than the depth threshold is defined as TP, a region where the ratio is less than the predetermined value and the depth is equal to or greater than the depth threshold is defined as FP, a region where the ratio is equal to or greater than the predetermined value and the depth is less than the depth threshold is defined as FN, and a region where the ratio is less than the predetermined value and the depth is less than the depth threshold is defined as TN. An image processing device wherein the inspection area determination unit determines the depth threshold based on an index obtained from (TP+TN) / (TP+FP+FN+TN), TP / (TP+FN), or TN / (FP+TN).

4. 3. The image processing device according to claim 2, The inspection area determination unit sets an approximation line for the distribution and determines the depth threshold as the depth on the approximation line corresponding to the size of the object at which the ratio is greater than or equal to the predetermined value.

5. 3. The image processing device according to claim 2, The inspection area determination unit is provided with a classification model that classifies an input image according to the distribution, sets an approximation line to the distribution for each classification of the input image by the classification model, and determines the depth threshold to be the depth corresponding to the size of the object at which the ratio is greater than or equal to the predetermined value on the approximation line set for the image classified by the classification model.

6. 1. An image processing method for an image processing device that determines an inspection area in an image of an object, in which the presence or absence of deterioration of the object is inspected, comprising: a step of estimating a depth, which is a distance in a depth direction in the image, for each pixel constituting the image; determining a depth threshold as a depth of a region of the object in the image where the ratio of the size of the object to the size of the deterioration to be inspected is equal to or greater than a predetermined value, and determining an area closer to the depth threshold as the inspection region based on the estimated depth of each pixel; and outputting the result of determining the inspection area.

7. A program that causes a computer to operate as the image processing device according to claim 1.

Images