Image processing device and image processing method

Abstract

To provide an image processing device and an image processing method for detecting an abnormality of a building member.SOLUTION: The present invention is an image processing device 100 for detecting an abnormality on a surface of a building member. The image processing device 100 includes a partitioning unit 1, an extraction unit 2, an estimation unit 3, and a calculation unit 4. The partitioning unit 1 partitions a learning image into a plurality of first patch images. The extraction unit 2 extracts a first feature quantity from each of the partitioned first patch images. The estimation unit 3 approximate estimates, with a density estimation method, a probability distribution to which the extracted first feature quantity follows. The partitioning unit 1 partitions an objective image into a plurality of second patch images. The extraction unit 2 extracts a second feature quantity from each of the partitioned second patch images. The calculation unit 4 calculates, by using the probability distribution obtained by the approximate estimation, an abnormality score of a region occupied by the second patch image from which the second feature quantity is extracted.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device and an image processing method. [Background technology]

[0002] In recent years, technological developments related to the preservation of buildings and the like using image analysis have been progressing, and related inventions have also been made public. For example, Patent Document 1 discloses an image processing method for extracting cracked areas from images of tunnel lining surfaces. According to the invention of Patent Document 1, cracked areas can be identified by discriminating them from images of the tunnel lining surface using feature values ​​selected for the cracked areas and a set threshold value or numerical range. In the invention of Patent Document 1, the average value of the angular second moment of the feature values ​​obtained from a histogram of pixel gray values ​​of the target area is used as the feature value. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2016-121953 A Summary of the Invention [Problem to be solved by the invention]

[0004] Patent Document 1 describes that a target region in which the average value of the angular second moment of a feature obtained from a histogram of pixel gray values ​​is equal to or greater than a predetermined threshold value (0.40) can be determined to be a crack region. However, in this type of crack determination using rule-based image analysis, a wide variety of rules must be applied, taking into consideration factors such as the material of the object being photographed and the photographing environment. For this reason, there are problems such as high personnel costs required for determining rules and limited scope of application. This problem is not limited to cracks, but also applies to abnormalities such as dirt and scratches. In addition, a method is known in which training data is created by annotating abnormal areas in an image and machine learning is performed. However, such a method requires a huge number of learning images, and therefore requires a great deal of effort to annotate abnormal areas. In addition, since the occurrence of abnormalities such as cracks is rare in the first place, it is difficult to collect a sufficient amount of image samples containing abnormalities. Due to these circumstances, there is a problem that it is not easy to determine abnormalities in building components using machine learning. From this perspective, an object of the present invention is to provide an image processing device and an image processing method for detecting abnormalities in building components. [Means for solving the problem]

[0005] The present invention, which solves the above-mentioned problems, is an image processing device for detecting abnormalities on the surface of a building component, the image processing device including: a first partitioning unit that partitions a learning image into a plurality of first patch images; a first extraction unit that extracts a first feature from each of the partitioned first patch images; an estimation unit that uses a density estimation method to approximately estimate a probability distribution to which the extracted first feature follows; a second partitioning unit that partitions a target image into a plurality of second patch images; a second extraction unit that extracts a second feature from each of the partitioned second patch images; and a calculation unit that uses the approximately estimated probability distribution to calculate an anomaly score of an area occupied by the second patch image from which the second feature has been extracted. The present invention also provides an image processing method in an image processing device for detecting abnormalities on the surface of a building component, the image processing method including a first partitioning step of partitioning a learning image into a plurality of first patch images, a first extraction step of extracting a first feature from each of the partitioned first patch images, an estimation step of approximately estimating a probability distribution to which the extracted first feature follows by a density estimation method, a second partitioning step of partitioning a target image into a plurality of second patch images, a second extraction step of extracting a second feature from each of the partitioned second patch images, and a calculation step of calculating, using the approximately estimated probability distribution, an anomaly score of an area occupied by the second patch image from which the second feature has been extracted.

[0006] If an image obtained by photographing the surface of a building component with no or few abnormalities is used as a learning image to extract a first patch image, a set of first feature values ​​extracted from an area with almost no abnormalities can be obtained. In addition, by applying a density estimation method to the set of first feature values ​​of the almost normal area, a probability distribution to which the feature values ​​of the area with no abnormalities follow can be approximately estimated. Therefore, by statistically evaluating how much the second feature value extracted from the target image obtained by photographing the target building component deviates from a typical normality based on the approximately estimated probability distribution, an abnormality judgment of the target building component can be appropriately performed. There are various types of feature values ​​(first feature value and second feature value) and they can be selected arbitrarily. Therefore, by selecting feature values ​​that do not depend on the material of the object to be photographed, the shooting environment, etc., the rule to be applied to the abnormality judgment can be easily selected, and the human cost required for determining the rule can be reduced, and the restriction on the application range can be relaxed or lifted. In addition, since normal image samples can be collected much more easily than abnormal image samples, if an image obtained by photographing a building component with no or few abnormalities is used as a learning image, it is easy to collect a sufficient amount of image samples for learning. Therefore, the feasibility of the present invention can be significantly improved.

[0007] It is preferable that the system further comprises a display control unit that displays the calculated abnormality scores in a heat map. This makes it easier to visualize detected abnormalities. It is also preferable that the learning image is "an image obtained by photographing the same type of building component as the building component photographed to obtain the target image," "an image obtained by photographing the building component photographed to obtain the target image when the building component was not in use," or "an image obtained by photographing a part of the building component photographed to obtain the target image that does not contain any abnormalities." This allows for easy collection of normal image samples. Effect of the Invention

[0008] According to the present invention, abnormalities in building components can be detected. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a functional configuration diagram of the image processing device according to the present embodiment. [Diagram 2] FIG. 1 is an explanatory diagram of patch extraction by raster scanning an image. [Diagram 3] This is an example of a graph when a bivariate Gaussian distribution is used for the probability model p(y;Θ). [Figure 4] FIG. 11 is an explanatory diagram of calculation of an abnormality score. [Diagram 5] 4 is a flowchart showing an image processing method according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiment of the present invention will be described in detail with reference to the drawings. Each drawing is merely a schematic illustration to allow a sufficient understanding of the present invention. Therefore, the present invention is not limited to the illustrated examples. In each drawing, common or similar components are denoted by the same reference numerals, and duplicated explanations thereof will be omitted. In this embodiment, the term "image" means image data unless otherwise specified.

[0011] FIG. 1 is a functional configuration diagram of an image processing device of this embodiment. The image processing device 100 is a computer that detects an abnormality on the surface of a building component. The image processing device 100 includes hardware such as an input unit, an output unit, a control unit, and a storage unit. For example, when the control unit is configured with a CPU (Central Processing Unit), information processing by a computer including the control unit is realized by program execution processing by the CPU. In addition, the storage unit included in the computer stores various programs for realizing the functions of the computer according to instructions from the CPU. This realizes collaboration between software and hardware. The programs can be provided by recording them on a recording medium or via a network. The image processing device 100 is connected to a console (not shown) used by a user so as to be able to communicate with the console. The console can display the processing results of the image processing device 100 and input requests to the image processing device 100.

[0012] The image processing device 100 includes a partition unit 1 (first partition unit, second partition unit), an extraction unit 2 (first extraction unit, second extraction unit), an estimation unit 3, a calculation unit 4, and a display control unit 5. The partitioning unit 1 partitions an image acquired by the image processing device 100 into a plurality of patch images. The image acquired by the image processing device 100 is, for example, an image generated by an imaging device photographing the surface of a building component. The images acquired by the image processing device 100 can be classified into learning images used for anomaly detection and target images to be used for determining whether or not there is an anomaly. The partitioning unit 1 partitions the learning image into a plurality of first patch images. Furthermore, the partitioning unit 1 partitions the target image into a plurality of second patch images. The extraction unit 2 extracts a feature amount from each of the patch images partitioned by the partition unit 1. The feature amount is, for example, a texture feature amount. The extraction unit 2 extracts a first feature amount from each of the first patch images. In addition, the extraction unit 2 extracts a second feature amount of the same type as the first feature amount from each of the second patch images. The estimation unit 3 approximately estimates a probability distribution to which the extracted first feature amount follows by using a density estimation method. The calculation unit 4 uses the approximately estimated probability distribution to calculate an anomaly score for the region occupied by the second patch image from which the second feature amount has been extracted. The display control unit 5 displays the calculated abnormality scores in a heat map. A console connected to the image processing device 100 can output the heat map on a screen.

[0013] (Acquisition of learning images) The learning images are image samples for detecting anomalies present in building components. For example, if the building components are concrete exterior materials, the learning images can be images of only concrete exterior materials. Also, if the building components are natural stone exterior materials, the learning images can be images of only a specific type of natural stone exterior material. It is preferable for the user to prepare a large number of learning images for each inspection object. It is preferable for each learning image to be an image obtained by photographing the surface of a building component that does not have anomalies or has few anomalies. In other words, it is assumed that "the learning images prepared in large quantities for each inspection object are a collection of almost normal image samples." Such an assumption is natural in light of the general situation that anomalies and defects are relatively few compared to the majority of the population, and in light of actual application examples of anomaly detection.

[0014] The learning images are preferably "images obtained by photographing the same type of building component as the building component photographed to obtain the target image," "images obtained by photographing the building component photographed to obtain the target image when the building component is not in use (including before the building is completed)," or "images obtained by photographing a part of the building component photographed to obtain the target image that does not have any abnormalities." It is difficult to collect a sufficient number of image samples that show abnormalities such as cracks, but by selecting learning images in this way, normal image samples can be easily collected.

[0015] (Creating patch images) For example, the partitioning unit 1 can partition an image acquired by the image processing device 100 into a plurality of patch images by two-dimensional raster scanning. Raster scanning is a method of first scanning a two-dimensional image one-dimensionally at points to obtain a scanning line, and then scanning the image in a direction perpendicular to the scanning line to obtain an image on a two-dimensional surface. FIG. 2 is an explanatory diagram of patch extraction by raster scanning an image. An image 10 is a learning image or a target image. The partitioning unit 1 prepares a local patch area 11 of, for example, k(px)×k(px) ("px": pixel) in the image 10. The partitioning unit 1 can create a patch image by scanning pixels in the local patch area 11. The partitioning unit 1 can move the local patch area 11 horizontally (main scanning direction) or vertically (sub-scanning direction) by a predetermined amount, and can create a patch image by scanning pixels in the moved local patch area 11. For example, as shown in FIG. 2, the partitioning unit 1 can create a patch image while moving the local patch area 11 horizontally by a predetermined amount, and then move the local patch area 11 vertically by the stride width s, and then create a patch image while moving it horizontally by a predetermined amount. As a result, the partitioning unit 1 can create multiple patch images for the image 10. Note that the multiple patch images created do not have to cover the entire image 10. A set X of first patch images created from a learning image is represented as X={I1, I2, . . . , I n The number of first patch images n created from one learning image can be any value. By making n smaller, the time required to calculate texture features (described later) can be shortened, and by making n larger, the accuracy of density estimation can be improved.

[0016] The value of k is preferably large enough to describe the surface structure of the image to be detected. The value of k can be changed appropriately depending on the object to be inspected, the shooting environment, and the texture feature amount, and is not limited to a specific value. The shape of the local patch area 11 is not limited to a square as shown in FIG. 2, but may be a rectangle. The local patch area 11 may be prepared by an operation of specifying a specific position and shape in the image using an image mask or a two-dimensional point spread function (PSF: Point Spread Function), etc. The process of creating the set of local patch regions 11 is not limited to the above raster scan, but may be, for example, random sampling using any two-dimensional probability distribution. In particular, a two-dimensional uniform distribution may be used as the two-dimensional probability distribution.

[0017] (Texture feature extraction) The first patch image, which is each element of the set X, can be regarded as a texture. The extraction unit 2 can convert the first patch image into a texture feature having the properties of a texture. The converted texture feature becomes the first feature. For example, the texture feature can be described as a fixed-length vector. As the texture feature, various features can be adopted. For example, texture image features such as first order statistics (FOS) when describing a pixel set constituting one patch image as a statistical summary, Halarick features based on Gray-Level Co-occurrence Matrix (GCLM), and Portilla-Simoncelli features based on Steerable Filter Pyramid, which is a multi-scale image decomposition method, may be used. In addition, the features extracted by the extraction unit 2 are not limited to texture features. For example, image features such as a feature map input to a fully connected layer in any convolutional neural network may be used. Alternatively, the features may be Bag-of-Visual-Words (BoVW) features based on SURF features or High-ordered Local Autocorrelation (HLAC) features.

[0018] Generally, texture features are m-dimensional real vectors (m is a natural number). The first patch image I i The first feature y converted from i , y i =f(I i )(f(I i ) is an element of the m-dimensional real space. By applying the transformation f to each element of the set X, the extraction unit 2 obtains a set of first features X′={f(I1), f(I2), . . . , f(I n )} can be obtained.

[0019] (Normal Distribution Modeling) According to the assumption that the large number of learning images prepared for each inspection target is a collection of almost all normal image samples, the proportion of areas containing abnormalities in the learning images is extremely small. i =f(I i ) is a probabilistic model y that has a high likelihood for texture features extracted from a patch image that does not contain any abnormalities. i It can be assumed that the first feature is expressed as p(y;Θ) (y is a random variable, and Θ is a parameter). The estimation unit 3 then uses a predetermined density estimation method to approximate the probability distribution that the first feature follows from the set X', thereby creating a probabilistic model p(y;Θ) that serves as a normal model. The density estimation method that can be used may be a parametric density estimation method or a non-parametric density estimation method. An example of a parametric density estimation method is a density estimation method that estimates a certain probability distribution, for example a multivariate Gaussian distribution N(μ,Λ -1 ) and its parameters, for example, in the case of a multivariate Gaussian distribution, the mean parameter μ (μ is an element of the m-dimensional real space) and the covariance parameter Λ -1 (Λ is a precision matrix, an element of the m×m dimensional real number space.) An example of a non-parametric density estimation method is the kernel density estimation method (KDE).

[0020] FIG. 3 is an example of a graph in which a bivariate Gaussian distribution is used for the probabilistic model p(y;Θ). For convenience of illustration, FIG. 3 corresponds to a visualization of a normal model in the case of m=2. The plots shown in FIG. 3 are attached to each first patch image. The positions of the plots correspond to the values ​​of each component of the two-dimensional first feature. Since the proportion of the area in which an abnormality exists in the learning image is extremely small, as shown in FIG. 3, the majority of the plots attached to the normal model are included inside a predetermined normal area C. The geometric shape and dimensions of the normal area C can be set appropriately.

[0021] (How anomaly scores are calculated) The above-mentioned probabilistic model p(y;Θ) has a high likelihood for the first feature extracted from normal patch images that occupy a majority of the set X of first patch images. The calculation unit 4 uses the probabilistic model p(y;Θ) to probabilistically calculate anomalies present in the building components to be inspected. Specifically, the partitioning unit 1 first partitions the target image to be inspected using a local patch region of k(px)×k(px) to create a plurality of second patch images. One of the created second patch images is designated as I * The second patch image I * can be regarded as a texture, the extraction unit 2 extracts the second patch image I * can be converted into an m-dimensional texture feature, and the second feature y * =f(I * The calculation unit 4 calculates the second feature amount y * =f(I * ) and the probability model p(y;Θ) that represents normality are measured by an appropriate scale to obtain the second patch image I * An anomaly score d can be calculated that indicates whether or not an anomaly exists in the area occupied by

[0022] The anomaly score d indicates a measure of the difference between the second patch image to be calculated and a normal model. For example, the calculation unit 4 calculates the measure by using a multivariate Gaussian distribution N(μ, Λ -1 ) and the second feature y * =f(I * ) and the Mahalanobis distance d = (f(I * )-μ) T Λ(f(I * )-μ). The calculation unit 4 also calculates the scale as the second feature quantity y * =f(I * ) negative log-likelihood d=-ln p(f(I *The scale can be calculated as follows: Θ ). The scale can be any real-valued function that can take small real values ​​for normal images without abnormalities and large real values ​​for images with abnormalities, and is not limited to a specific function.

[0023] FIG. 4 is an explanatory diagram of the calculation of the abnormality score. The image 20 is a target image when the surface of concrete is photographed, and a crack 21 is present. The graph 30 corresponds to a plan view of the graph in FIG. 3. The partitioning unit 1 can create, from the image 20, a second patch image 22a (shown by a rectangular dashed line) in which the crack 21 is not present, and a second patch image 22b (shown by a rectangular two-dot chain line) in which the crack 21 is present. The extraction unit 2 extracts second feature amounts from the second patch images 22a and 22b. For each of the second patch images 22a and 22b, a plot corresponding to the extracted second feature amount is added to the graph 30. The abnormality score d can be evaluated as the deviation of the plot from the normal region C.

[0024] As shown in FIG. 4, the plot added to graph 30 for second patch image 22a is included inside normal region C. Therefore, the abnormality score d for second patch image 22a is small. This indicates that it is extremely unlikely that an abnormality exists at the position of second patch image 22a. On the other hand, the plot added to graph 30 for second patch image 22b deviates outside normal region C. Therefore, the abnormality score d for second patch image 22b is large. This indicates that it is extremely likely that an abnormality exists at the position of second patch image 22b.

[0025] (Displaying points of interest for the input image) The calculation unit 4 calculates an anomaly score d for each of the multiple second patch images created by the partition unit 1 from the target image. The display control unit 5 creates a heat map in which the magnitude of the calculated anomaly score d corresponds to a color, and can output the heat map to a screen. For example, the display control unit 5 can display the target image and the heat map side by side, or display the heat map superimposed on the target image. Such a display can easily visualize the detected anomaly, promote line of sight guidance for the worker, and improve work efficiency. It can also prevent anomalies from being overlooked.

[0026] [process] The image processing method of this embodiment will be described. FIG. 5 is a flowchart showing the image processing method of this embodiment. First, the image processing device 100 acquires a learning image obtained by photographing a specific building component (step S1). Next, the partitioning unit 1 partitions the learning image into a plurality of first patch images (step S2). Next, the extraction unit 2 extracts a first feature amount from each of the first patch images (step S3). Next, the estimation unit 3 approximately estimates the probability distribution to which the first feature amount follows by a density estimation method (step S4). A probability model p(y;Θ) is created by the approximate estimation. Next, the image processing device 100 acquires a target image obtained by photographing the same building component (step S5). Next, the partitioning unit 1 partitions the target image into a plurality of second patch images (step S6). Next, the extraction unit 2 extracts a second feature amount from each of the second patch images (step S7). Next, the calculation unit 4 calculates an anomaly score of an area occupied by the second patch image from which the second feature amount is extracted (step S8). Next, the display control unit 5 displays the calculated anomaly scores in a heat map (step S9). This completes the image processing method of this embodiment. Note that steps S1 to S4 correspond to a learning phase. Steps S5 to S9 correspond to a recognition phase.

[0027] [effect] According to this embodiment, the first patch image is extracted from the image obtained by photographing the surface of the building component with no or few abnormalities as a learning image, so that a set of first feature values ​​of the area with no abnormalities is obtained. In addition, by applying a density estimation method to the set of first feature values ​​of the normal area, the probability distribution to which the feature values ​​of the area with no abnormalities follow can be approximately estimated. Therefore, the second feature value extracted from the target image obtained by photographing the target building component is statistically evaluated to what extent it deviates from typical normality based on the approximately estimated probability distribution, so that an abnormality judgment of the target building component can be appropriately performed. There are various types of feature values ​​(first feature value and second feature value) and they can be selected arbitrarily. Therefore, by selecting feature values ​​that do not depend on the material of the photographed object, the photographing environment, etc., the rule to be applied to the abnormality judgment can be easily selected, and the human cost required for determining the rule can be reduced, and the restriction of the application range can be relaxed or lifted. In addition, since normal image samples can be collected much more easily than abnormal image samples, it is easy to collect a sufficient amount of image samples for learning by using images obtained by photographing building components with no or few abnormalities as learning images. Therefore, the feasibility of the present invention can be significantly improved.

[0028] In addition, in this embodiment, it is possible to present candidate regions for anomalies without using annotations of anomalies that were used in conventional machine learning. Therefore, a large amount of detection position annotations in advance, which was a major barrier to the introduction of conventional image inspection support AI, is not required, making it easier to implement anomaly detection. Accordingly, this embodiment can be applied to new inspection cases for which there are no past inspection cases. In addition, the image processing device 100 of this embodiment can be implemented as a standalone computer to realize on-site operation. In addition, there is no increase in personnel costs associated with the introduction of AI to improve business efficiency. Furthermore, by introducing the image processing method of this embodiment into the screening stage of anomaly detection, building materials with no or few anomalies can be excluded from the inspection target from the beginning, narrowing down the inspection target. This reduces the personnel costs involved in the anomaly detection work. In addition, it is possible to significantly reduce costs and labor hours in cases where tens of thousands of images and several months of work are required, such as the complete inspection of exterior materials of a large-scale building.

[0029] [Variations] (a): The present invention is not limited to detecting cracks, but can also detect dirt, scratches, and other abnormalities. (b): In creating the patch images for section 1, each patch image may be sampled from overlapping pixels, i.e. a set of patch images is created such that the same pixel in the image is scanned multiple times. (c): In this embodiment, the case of detecting abnormalities (cracks) on the surface of a building component has been described. However, in the present invention, if a two-dimensional image can be acquired by a predetermined observation operation and image processing can be performed, abnormalities can be detected in the same framework as described in this embodiment. For example, a learning image and a target image are prepared for an image of the inside of an exterior material photographed by an infrared camera, and abnormalities such as exterior material lifting can be detected in the target image. (d): When the first building component from which the learning image was obtained and the second building component from which the target image was obtained are of the same type, the anomaly detection of the present invention can be applied even if the materials used for the first and second building components are different. In other words, the same learning image can be reused for multiple types of second building components made of different materials. In this case, it is preferable that the surface properties (texture) of the materials used for the first and second building components are the same or similar.

[0030] (e) It is also possible to realize a technology that appropriately combines the various technologies described in this embodiment. (f): The software described in this embodiment can be realized as hardware, and the hardware can be realized as software. (g) In addition, the hardware, software, flow charts, etc. may be modified as appropriate without departing from the spirit and scope of the present invention. [Explanation of symbols]

[0031] 100 Image processing device 1 Partition (1st partition, 2nd partition) 2 Extraction part (1st extraction part, 2nd extraction part) 3 Estimation part 4 Calculation section 5. Display control section

Claims

1. An image processing device for detecting an abnormality in a building component, A first partitioning unit that partitions the learning image into a plurality of first patch images; a first extraction unit that extracts a first feature amount from each of the partitioned first patch images; an estimation unit that estimates approximately a probability distribution to which the extracted first feature value follows by a density estimation method; a second partitioning unit that partitions the target image into a plurality of second patch images; a second extraction unit that extracts a second feature amount from each of the partitioned second patch images; a calculation unit that calculates, using the approximately estimated probability distribution, an anomaly score for an area occupied by the second patch image from which the second feature amount is extracted.

2. The image processing device according to claim 1 , further comprising a display control unit that displays the calculated anomaly scores in a heat map.

3. The learning image is The image is obtained by photographing a building component of the same type as the building component photographed to obtain the target image. The image of the building component photographed to obtain the target image is obtained by photographing the building component when the building component is not in use.

3. The image processing apparatus according to claim 1, wherein the image is an image obtained by photographing a portion of the building member that does not contain an abnormality, among the building members photographed to obtain the target image.

4. An image processing method for detecting an abnormality in a building component, comprising: A first partitioning step of partitioning the learning image into a plurality of first patch images; a first extraction step of extracting a first feature amount from each of the partitioned first patch images; an estimating step of approximating a probability distribution to which the extracted first feature quantity follows by a density estimation method; a second partitioning step of partitioning the target image into a plurality of second patch images; a second extraction step of extracting a second feature amount from each of the partitioned second patch images; and calculating, using the approximately estimated probability distribution, an anomaly score for an area occupied by the second patch image from which the second feature amount is extracted.

Images