Image processing device, image processing method, and image processing program
The image processing device efficiently measures damage sizes on structures by generating 3D models and correcting measurements based on camera and normal vector angles, addressing inefficiencies in existing techniques.
Patent Information
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- FUJIFILM CORP
- Filing Date
- 2022-03-10
- Publication Date
- 2026-06-15
Smart Images
Figure 0007874088000001 
Figure 0007874088000002 
Figure 0007874088000003
Abstract
Description
【Technical Field】 【0001】 The present invention relates to an image processing apparatus, an image processing method, and an image processing program, and more particularly to an image processing apparatus, an image processing method, and an image processing program that process an image of a structure to detect damage on the surface of the structure and measure its size. 【Background Art】 【0002】 Social infrastructure structures such as bridges, dams, and tunnels are required to be inspected regularly once every few years. For example, bridges are required to be inspected regularly once every five years. As a technology to support this work, a technology for automatically extracting damage (such as cracks and free lime) that appears on the surface of a structure from an image of the structure is known. 【0003】 Patent Document 1 describes a technique for detecting appearance deterioration such as cracks from an image of a structure (erected concrete pole) and measuring its size (length and width), and describes correcting the measurement result according to the position of the appearance deterioration in order to obtain a more accurate size. 【Prior Art Documents】 【Patent Documents】 【0004】 【Patent Document 1】 Japanese Patent Application Laid-Open No. 2002-162260 【Summary of the Invention】 【Problems to be Solved by the Invention】 【0005】 However, the technique described in Patent Document 1 has a drawback that in order to obtain information for correcting the measurement result, it is necessary to measure the position, angle, etc. of the appearance deterioration in detail, which requires a great deal of labor for measurement. 【0006】 This invention has been made in view of these circumstances, and aims to provide an image processing device, an image processing method, and an image processing program that can easily measure the precise size of damage. [Means for solving the problem] 【0007】 (1) An image processing device comprising a processor, the processor performing the following processes: acquiring multiple images of a subject taken from multiple viewpoints with overlapping shooting ranges using a camera; detecting damage to the surface of the subject from the acquired multiple images; analyzing the acquired multiple images to generate 3D point cloud data of feature points; generating a 3D patch model of the subject based on the generated point cloud data; selecting an image corresponding to each patch of the 3D patch model from the acquired multiple images; measuring the size of the damage in the area corresponding to the patch in the selected image; generating correction information necessary for correcting the measurement result of the damage size based on a camera vector a connecting the camera position and the patch when the selected image was taken, and a normal vector b of the patch; and correcting the measurement result of the damage size based on the generated correction information. 【0008】 (2) The image processing device of (1), wherein the processor generates correction information by calculating the cosine value cosθ of the angle θ between the camera vector a and the normal vector b in the process of generating correction information. 【0009】 (3) The processor is the image processing device of (2) that calculates the cosine value cosθ using cosθ = a·b / |a||b|. 【0010】 (4) The image processing apparatus according to (2) or (3), wherein the processor corrects the measurement result of the size of the damage by dividing the measurement result of the size of the damage by the cosine value cosθ in the process of correcting the measurement result of the size of the damage. 【0011】 (5) The processor is an image processing device, one of (1) to (4), which, in the process of selecting an image corresponding to a patch, selects the image in which the angle θ between the camera vector a and the normal vector b is smallest. 【0012】 (6) The processor is one of the image processing devices (1) to (4) that selects the image with the highest capture resolution in the process of selecting an image corresponding to a patch. 【0013】 (7) The processor is an image processing device, one of (1) to (4), which, in the process of selecting an image corresponding to a patch, selects the image in which the location of the region corresponding to the patch is closest to the center of the image. 【0014】 (8) The processor is an image processing device, any one of (1) to (7), which generates a 3D patch model by generating patches using triangular meshes or quadrilateral meshes in the process of generating a 3D patch model. 【0015】 (9) The processor further performs the process of calculating the physical dimensions of the patches in the three-dimensional patch model, an image processing device from any one of (1) to (8). 【0016】 (10) The processor is an image processing device of (9) that calculates the physical dimensions of a patch based on distance measurement information associated with the image in the process of calculating the physical dimensions of the patch. 【0017】 (11) The processor is an image processing device of (9) that calculates the physical dimensions of a patch based on reference point information contained in the image, in the process of calculating the physical dimensions of a patch. 【0018】 (12) An image processing apparatus, any one of (1) to (11), wherein the damage is a crack, and the processor measures the width and / or length of the crack in a process of measuring the size of the damage. 【0019】 (13) The image processing device of (12), wherein the processor extracts patches of continuous cracks, and further performs the process of measuring the length of the continuous cracks by summing the corrected crack lengths obtained for each extracted patch. 【0020】 (14) An image processing method comprising the steps of: acquiring multiple images of a subject taken from multiple viewpoints with overlapping shooting ranges using a camera; detecting damage to the surface of the subject from the acquired multiple images; analyzing the acquired multiple images to generate 3D point cloud data of feature points; generating a 3D patch model of the subject based on the generated point cloud data; selecting an image from the acquired multiple images that corresponds to each patch of the 3D patch model; measuring the size of the damage in the region corresponding to the patch in the selected image; generating correction information necessary for correcting the measurement result of the damage size based on a camera vector a connecting the camera position when the selected image was taken and the patch, and a normal vector b of the patch; and correcting the measurement result of the damage size based on the generated correction information. 【0021】 (15) An image processing program that enables a computer to perform the following functions: (15) Acquire multiple images of a subject from multiple viewpoints with overlapping shooting ranges using a camera; detect damage to the surface of the subject from the acquired multiple images; analyze the acquired multiple images to generate 3D point cloud data of feature points; generate a 3D patch model of the subject based on the generated point cloud data; select an image corresponding to each patch of the 3D patch model from the acquired multiple images; measure the size of the damage in the area corresponding to the patch in the selected image; generate correction information necessary for correcting the measurement result of the damage size based on a camera vector a connecting the camera position and the patch when the selected image was taken, and a normal vector b of the patch; and correct the measurement result of the damage size based on the generated correction information. [Effects of the Invention] 【0022】 According to the present invention, the exact size of damage can be easily measured. 【Brief Description of Drawings】 【0023】 [Figure 1] Figure showing the schematic configuration of a system for measuring the width of cracks [Figure 2] Conceptual diagram of photography [Figure 3] Block diagram showing an example of the hardware configuration of an image processing apparatus [Figure 4] Block diagram of the main functions of an image processing apparatus [Figure 5] Figure showing an overview of SfM [Figure 6] Flowchart showing an overview of the procedure of SfM processing [Figure 7] Figure showing an example of a TIN model [Figure 8] Figure showing an example of a three-dimensional shape model [Figure 9] Conceptual diagram of crack width correction [Figure 10] Conceptual diagram of calculation of correction value [Figure 11] Flowchart showing the procedure of image processing by an image processing apparatus [Figure 12] Block diagram of the main functions of an image processing apparatus [Figure 13] Conceptual diagram of image selection [Figure 14] Flowchart showing the procedure of image selection processing in an image selection unit [Figure 15] Conceptual diagram of a method for measuring the length of a crack [Figure 16] Flowchart showing the procedure of processing when measuring the size of damage without correction 【Modes for Carrying Out the Invention】 【0024】 Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. 【0025】 [First Embodiment] This section explains the process using the example of measuring the width of a crack on the surface of a bridge pier. A bridge pier is an example of a concrete structure, particularly one made of reinforced concrete. A crack is an example of damage. The width of the crack is an example of the size of the damage. 【0026】 [System Configuration] Figure 1 shows a schematic configuration of a system for measuring the width of cracks. 【0027】 As shown in the figure, the system 1 of this embodiment includes a camera 10 and an image processing device 100 that processes images captured by the camera 10. 【0028】 Camera 10 is used to photograph the object to be measured. A general-purpose digital camera (including those mounted on mobile devices, etc.) is used as camera 10. As described above, in this embodiment, the object to be measured is a crack that has occurred on the surface of the bridge pier Ob. In this case, the bridge pier Ob is used as the subject for photography. 【0029】 Figure 2 is a conceptual diagram of the shooting process. 【0030】 In the figure, the rectangular area indicated by the symbol R represents the range of a single image capture. As shown in Figure 2, the image capture is performed by overlapping the capture range R from multiple viewpoints. As an example, images are captured with an overlap OL of 80% or more and a sidelap SL of 60% or more. Overlap OL refers to the overlap in the direction of the course when capturing along a straight course. Therefore, "overlap OL of 80% or more" means that the overlap rate of images in the direction of the course is 80% or more. Sidelap SL refers to the overlap between courses. Therefore, "sidelap SL of 60% or more" means that the overlap rate of images between courses is 60% or more. 【0031】 The image processing device 100 processes images captured by the camera 10 to detect cracks occurring on the surface of a structure and to measure their width. The image processing device 100 is composed of a computer equipped with an input device and a display device. 【0032】 Figure 3 is a block diagram showing an example of the hardware configuration of an image processing device. 【0033】 As shown in the figure, the image processing device 100 is comprised of a CPU (Central Processing Unit) 101, RAM (Random Access Memory) 102, ROM (Read Only Memory) 103, auxiliary storage device 104, communication interface (Interface, IF) 105, input device 106, and display device 107, etc. The ROM 103 and / or auxiliary storage device 104 store programs executed by the CPU 101 and various data. The auxiliary storage device 104 is comprised of, for example, an HDD (Hard Disk Drive) or SSD (Solid State Drive). The input device 106 is comprised of, for example, a keyboard, mouse, or touch panel. The display device 107 is comprised of, for example, a liquid crystal display (Liquid Crystal Display) or an organic light-emitting diode display (Organic Light Emitting Diode Display). Images captured by the camera 10 are taken into the image processing device 100 via the communication interface 105 and stored in the auxiliary storage device 104. 【0034】 Figure 4 is a block diagram of the main functions of the image processing device. 【0035】 As shown in the figure, the image processing device 100 mainly has functions such as an image acquisition unit 111, a point cloud data generation unit 112, a 3D patch model generation unit 113, a 3D shape model generation unit 114, a crack detection unit 115, a crack measurement unit 116, a correction value calculation unit 117, and a measurement result correction unit 118. These functions are realized by the processor executing a predetermined program (image processing program). 【0036】 The image acquisition unit 111 performs the process of acquiring the image group IG to be processed. The image group IG to be processed consists of multiple images taken by the camera 10 from multiple viewpoints with overlapping shooting ranges R. The image acquisition unit 111 acquires the image group IG to be processed from the auxiliary storage device 104. 【0037】 The point cloud data generation unit 112 analyzes the image group IG acquired by the image acquisition unit 111 and performs processing to generate three-dimensional point cloud data of feature points. In this embodiment, this processing is performed using SfM (Structure from Motion) and MVS (Multi-View Stereo) technologies. 【0038】 Figure 5 shows an overview of SfM. 【0039】 SfM is a technique that estimates the position and orientation of an image captured by a camera, and also performs 3D reconstruction of feature points. The SfM technique itself is a well-known technology. The general outline of its processing is as follows: 【0040】 Figure 6 is a flowchart illustrating the general procedure for SfM processing. 【0041】 First, multiple images (image group) to be processed are acquired (Step S1). Next, feature points are detected from each acquired image (Step S2). Next, by comparing the feature points of two image pairs, matching feature points are detected as corresponding points (Step S3). In other words, feature point matching is performed. Next, from the detected corresponding points, the camera parameters of the camera that took the two image pairs (e.g., base matrix, fundamental matrix, and internal parameters, etc.) are estimated (Step S4). Next, based on the estimated camera parameters, the shooting position and orientation are estimated (Step S5). In addition, the 3D position of the feature points of the object is determined (Step S6). In other words, 3D reconstruction of the feature points is performed. After this, bundle adjustment is performed as needed. That is, the 3D point cloud, which is the set of the above feature points in 3D coordinates, is adjusted so that the reprojection error of the point cloud to the camera is minimized. coordinate The camera's internal parameters (focal length, principal point) and external parameters (position, rotation) are adjusted. 【0042】 The 3D points reconstructed by SfM are unique and sparse. Typical 3D models consist mostly of low-feature textures (e.g., walls). MVS attempts to reconstruct these low-feature 3D textures. MVS generates a dense point cloud using the "imaging position and orientation" estimated by SfM. The MVS technique itself is well-known; therefore, a detailed explanation will be omitted. 【0043】 Furthermore, the reconstructed shape and shooting position obtained by SfM are point clouds represented by dimensionless coordinate values. Therefore, the shape cannot be quantitatively understood in its current form. Thus, it is necessary to provide physical dimensions (actual dimensions). Known techniques are employed for this process. For example, techniques such as extracting reference points (e.g., ground control points) from the image and assigning physical dimensions can be employed. Ground control points (GCPs) are landmarks that contain geospatial information (latitude, longitude, altitude) that can be seen within the captured image. Therefore, in this case, it is necessary to set reference points at the shooting stage. Also, if distance measurement information is associated with the captured image, physical dimensions can be assigned using that distance measurement information. For example, when photographing a subject using an unmanned aerial vehicle (UAV) such as a drone, the camera... and When using LIDAR (Light Detection and Range or Laser Imaging Detection and Range) mounted on an unmanned aerial vehicle for imaging, ranging information from the LIDAR can be acquired along with the image. This ranging information can be used to add physical dimension information to the 3D point cloud data obtained by SfM. In addition, it is possible to extract objects with known physical dimensions from the image and assign those dimensions to them. 【0044】 The 3D patch model generation unit 113 generates a 3D patch model of the subject based on the 3D point cloud data of the subject generated by the point cloud data generation unit 112. Specifically, it generates patches (meshes) from the generated 3D point cloud and generates a 3D patch model. This makes it possible to represent the surface relief with a small number of points. This process is performed using known techniques such as 3D Delaunay triangulation. Therefore, a detailed explanation is omitted. As an example, in this embodiment, a TIN (Triangular Irregular Network) model is generated using 3D Delaunay triangulation. The TIN model is an example of a 3D patch model. 【0045】 Figure 7 shows an example of a TIN model. 【0046】 As shown in the figure, in the TIN model, the surface is represented by a collection of triangles. In other words, patch P is generated by a triangular mesh. 【0047】 The 3D shape model generation unit 114 performs texture mapping on the 3D patch model generated by the 3D patch model generation unit 113 to generate a 3D shape model with a texture applied. This process is performed by interpolating the space within each patch of the 3D patch model with captured images. As described above, in the image processing device 100 of this embodiment, SfM and MVS processing are performed in the point cloud data generation unit 112. Through this SfM and MVS processing, images of the region corresponding to each patch and the corresponding positions within those images can be determined. Therefore, if the vertices of the generated surface can be observed, the texture to be applied to that surface can be associated with them. The 3D shape model generation unit 114 selects images corresponding to each patch and extracts images of the region corresponding to the patch as a texture from the selected images. Specifically, it projects the vertices of the patch onto the selected image and extracts images of the region enclosed by the projected vertices as a texture. The extracted texture is applied to the patch to generate a 3D shape model. That is, the space within the patch is interpolated with the extracted texture to generate a 3D shape model. 【0048】 Figure 8 shows an example of a three-dimensional shape model. 【0049】 As shown in the figure, by applying a texture to each patch, color information is added to each patch. If cracks exist in the subject, crack C will be displayed at the corresponding location. 【0050】 The generated 3D shape model is stored in an auxiliary storage device 104 or the like, as needed. It is also displayed on the display device 107, as needed. 【0051】 The crack detection unit 115 performs a process to detect cracks from each acquired image. Known techniques can be used for detecting cracks from images. For example, techniques that use image recognition models generated by machine learning, deep learning, etc., to detect cracks from images can be employed. The type of machine learning algorithm is not particularly limited. For example, algorithms using neural networks such as RNN (Recurrent Neural Network), CNN (Convolutional Neural Network), and MLP (Multilayer Perceptron) can be used. When detecting cracks using an image recognition model, preprocessing of the image is performed as necessary. Chi Then, processing (such as filtering) is applied to improve recognition accuracy. 【0052】 The crack measurement unit 116 performs a process to measure the width of the cracks detected by the crack detection unit 115. Known image measurement techniques are used for the measurement. If the texture applied to each patch contains cracks, the width of those cracks is measured by this process. 【0053】 The correction value calculation unit 117 performs a process to calculate a correction value. This correction value is used to correct the measured width of the crack in a patch that contains a crack. 【0054】 Here, we will explain the need to correct the measured values of the crack width. 【0055】 Figure 9 is a conceptual diagram of crack width correction. 【0056】 As shown in the figure, when the camera 10 is positioned directly in front of the surface S containing the crack C and an image is taken, the width of the crack measured from that image matches the actual width CW of the crack. 【0057】 On the other hand, when the camera 10 is tilted to capture an image of a surface S containing a crack C, that is, when the image is captured in an upward-angled position, the crack width CWx measured from that image will be smaller than the actual crack width CW due to the upward angle (CWx <CW)。 【0058】 Therefore, for cracks whose width was measured from an image taken at an angle, it is necessary to correct the measured width. 【0059】 If the angle of tilt is θ, then the actual crack width CW and the crack measured from the image taken in the tilted position are... Width C The following relationship holds between Wx and the following: 【0060】 CW = CWx / cosθ Therefore, the correction value calculation unit 117 calculates cosθ as the correction value σ. The correction value σ is an example of correction information. 【0061】 As described above, in the image processing device 100 of this embodiment, point cloud data is generated by SfM in the point cloud data generation unit 112. SfM can estimate the position (camera position) and orientation at which the image was captured. Therefore, correction values can be calculated using this information. 【0062】 Figure 10 is a conceptual diagram of how the correction value is calculated. 【0063】 The figure shows an example of how to calculate the correction value σ in patch Pi, which is indicated by the shaded area. 【0064】 Image Ii is the image obtained by extracting the texture to be applied to the patch Pi that is to be calculated. Image Ii is the image corresponding to patch Pi. Position Pc is the camera position (shooting position) when this image Ii was taken. The camera position Pc can be estimated using SfM. 【0065】 Let the camera vector be a, which is the vector connecting the camera's position Pc and the centroid or center of patch Pi. Let b be the normal vector of patch Pi. 【0066】 The tilt angle θ when image II was captured is determined by the angle between the camera vector a and the normal vector b. The cosine value cosθ of this angle θ can be calculated using the formula cosθ = a·b / |a||b|. Therefore, the correction value σ can be calculated using the formula σ = a·b / |a||b|. 【0067】 The correction value calculation unit 117 calculates a correction value σ for each patch. Alternatively, the system may be configured to calculate the correction value σ only for patches where the applied texture contains cracks. 【0068】 The measurement result correction unit 118 corrects the width of the cracks measured from the image in patches containing cracks using the correction value σ calculated by the correction value calculation unit 117. That is, it corrects the measured value CWx using CW = CWx / σ to calculate the actual crack width CW. This allows for obtaining accurate information on the crack width. 【0069】 The calculated information on the actual crack width is stored in the auxiliary storage device 104, etc., as needed. It is also displayed on the display device 107 as needed. When storing the information, it is stored in association with the 3D shape model, or in association with the captured image. 【0070】 [Image processing procedure (image processing method)] Figure 11 is a flowchart showing the image processing procedure performed by an image processing device. 【0071】 First, a process is performed to acquire a set of images of the subject (step S11). In this embodiment, a set of images of the bridge piers is acquired. As described above, the shooting is performed by overlapping the shooting range from multiple viewpoints. 【0072】 Next, a process is performed to detect damage from each acquired image (step S12). In this embodiment, cracks are detected. Each crack detected from each image is assigned identification information and recorded. 【0073】 Next, a process is performed to measure the size of the damage detected in each image (step S13). In this embodiment, the width of the crack is measured. The measured width information is recorded in association with the identification information of the crack that was measured. 【0074】 Next, a process is performed to generate 3D point cloud data of the subject from the acquired image set (step S14). As described above, this process is performed using SfM and MVS. In addition to 3D point cloud data of the subject, SfM and MVS can obtain information on the camera's position and orientation when each image was taken. 【0075】 Next, a process is performed to generate a 3D patch model of the subject based on the generated 3D point cloud data of the subject (step S15). In this embodiment, a TIN model is generated as the 3D patch model using a 3D Delaunay triangulation (see Figure 7). 【0076】 Next, texture mapping is performed on the generated 3D patch model to generate a 3D shape model of the subject (step S16). As described above, in this process, the image of the region corresponding to each patch is extracted as a texture from the image corresponding to each patch of the 3D patch model. Then, the extracted texture is applied to the patch to generate the 3D shape model (see Figure 8). 【0077】 Next, a process is performed to calculate the correction value σ (step S17). This process is performed for at least the patch containing the damage. The correction value σ is calculated from the tilt angle θ when the image corresponding to the patch is taken, as described above. This angle θ is obtained from the angle between the camera vector a and the normal vector b. The correction value σ is obtained from the cosine value cosθ of this angle θ. That is, it is obtained by σ = cosθ = a·b / |a||b|. 【0078】 Next, for patches containing damage, a process is performed to correct the measured value of the damage with a correction value (step S18). That is, the measured value of the damage is divided by the correction value σ to calculate the actual size of the damage. In this embodiment, the measured value of the crack width CWx is divided by the correction value σ to calculate the actual width CW. That is, the actual width CW is calculated using CW = CWx / σ. The correction is performed for all patches containing damage. This makes it possible to determine the accurate width of the crack. 【0079】 As described above, the image processing device 100 of this embodiment can eliminate the effects of tilt and accurately measure the size of damage. Furthermore, since there is no need to measure the tilt angle during shooting, on-site shooting and other operations can be carried out smoothly. 【0080】 In this embodiment, a 3D shape model is generated, but if the sole purpose is to measure the size of the damage, it is not necessarily required to generate a 3D shape model. At the very least, it is sufficient to identify the image corresponding to each patch. 【0081】 [Second Embodiment] As described above, the 3D shape model is generated by interpolating each patch of the 3D patch model with a texture. The texture applied to each patch is generated by extracting it from the image corresponding to each patch. There may be multiple such images. That is, since the shooting is performed from multiple viewpoints with overlapping shooting ranges, there may be multiple images corresponding to a single patch. When there are multiple corresponding images, the question arises as to which image to select. The image processing device of this embodiment selects an image from the following perspective: that is, it selects the image most suitable for measuring damage as the image corresponding to the patch. 【0082】 Figure 12 is a block diagram of the main functions of the image processing device in this embodiment. 【0083】 As shown in the figure, the 3D shape model generation unit 114 differs from the image processing apparatus 100 of the first embodiment in that it has the function of the image selection unit 114A. The function of the image selection unit 114A will be described below. 【0084】 The image selection unit 114A performs the process of selecting one image from among multiple images that correspond to each patch. The selection is made from a measurement perspective, and the image most suitable for measuring damage is selected as the image corresponding to the patch. In this embodiment, the image with the smallest angle θ between the camera vector a and the normal vector b is selected. In other words, the image with the smallest tilt angle is selected. This reduces the effect of tilt, allows for more accurate measurement of damage, and enables more accurate correction of the measured values. 【0085】 Figure 13 is a conceptual diagram of image selection. 【0086】 The figure shows an example where there are two images corresponding to patch Pi, indicated by the diagonal lines. Let one image be the first image I1i and the other be the second image I2i. 【0087】 When photographing the first image I1i, the position of the camera is the first position Pc1, and the camera vector of the first image I1i is the first camera vector a1. Also, when photographing the second image I2i, the position of the camera is the second position Pc2, and the camera vector of the second image I2i is the second camera vector a2. 【0088】 The first camera vector a1 is defined as the vector connecting the first position Pc1 and the centroid or center of the patch Pi. Also, the second camera vector a2 is defined as the vector connecting the second position Pc2 and the centroid or center of the patch Pi. 【0089】 For the images, the one with a smaller angle between the camera vector and the normal vector is selected. For example, let the angle between the first camera vector a1 and the normal vector b be θ1, and the angle between the second camera vector a2 and the normal vector b be θ2. When θ1 > θ2, the second image I2i is selected as the image corresponding to the patch Pi. 【0090】 In the case of θ1 > θ2 (0 ≤ θ1 ≤ 90°, 0 ≤ θ2 ≤ 90°), cosθ1 < cosθ2. Therefore, the image selection unit 114A calculates the cosine value (cosθ) of the angle between the camera vector and the normal vector, and selects the image with the largest such value as the image corresponding to the patch. 【0091】 FIG. 14 is a flowchart showing the procedure of the image selection process in the image selection unit. 【0092】 First, a process of obtaining an image corresponding to the patch is performed (step S21). Next, it is determined whether there are multiple acquired images mosquitoA process to determine if an image corresponds to a patch is performed (step S22). If there are no multiple images corresponding to a patch, the selection process ends. On the other hand, if there are multiple images corresponding to a patch, a process to select an image is performed. First, a process to calculate the angle between the camera vector and the normal vector of each image is performed (step S23). This process is performed by finding the cosine value (cosθ) of the angle between the camera vector and the normal vector. Next, based on the calculation result, a process to select the image with the smallest angle between the camera vector and the normal vector is performed (step S24). The selected image is designated as the image corresponding to the patch. 【0093】 As described above, the image selected in this process is the one with the smallest tilt angle. This reduces the effect of tilt, allows for more accurate measurement of the size of the damage, and enables accurate correction of those measurements. 【0094】 The selected images are used for texture extraction. If the extracted textures contain cracks, they are subject to measurement, and the measured values are then corrected. 【0095】 [Differentiation] In the above embodiment, we described an example where the image corresponding to the patch is selected based on the angle between the camera vector and the normal vector. However, the method for selecting the image corresponding to the patch is not limited to this. Other examples of methods for selecting the image corresponding to the patch are described below. 【0096】 (1) Method of selection based on shooting resolution If multiple images exist that correspond to a patch, the image with the highest capture resolution will be selected as the image corresponding to the patch. 【0097】 Shooting resolution is synonymous with resolution. Therefore, for the same subject, the image with higher resolution will be selected as the image corresponding to the patch. For example, in the case of a crack, the image in which the crack is captured with higher resolution will be selected as the image corresponding to the patch. When the same subject is photographed with the same camera, images taken at a closer position to the subject will have higher resolution. In other words, images taken at a shorter shooting distance will have higher resolution. Therefore, in this case, the image taken at the closest shooting distance, i.e., the image in which the camera position is closest to the patch, will be selected as the image corresponding to the patch. 【0098】 (2) Selection method based on the location of the area corresponding to the patch If multiple images correspond to a patch, the image whose location is closest to the center of the image corresponding to the patch will be selected as the image corresponding to the patch. 【0099】 Generally, camera lenses exhibit greater aberrations the further they are from the optical axis (center). Consequently, the quality of the captured image also deteriorates the further it is from the center. Specifically, distortions and other distortions become more pronounced. 【0100】 Therefore, if multiple images correspond to a patch, the image whose location is closest to the center of the image corresponding to the patch is selected as the image corresponding to the patch. In this case, for example, for each image, the distance between the centroid or center of the area corresponding to the patch and the center of the image is calculated, and the image with the smallest calculated distance is selected as the image corresponding to the patch. 【0101】 This allows for the measurement of damage size based on high-resolution images. As a result, damage size can be measured with greater accuracy. 【0102】 (3) Other methods It is also possible to combine multiple of the above methods to select images. 【0103】 For example, a method can be adopted that assigns a priority to each selection method. In this case, if an image cannot be selected using a higher-priority selection method, a lower-priority selection method is used. For example, the first selection method may be based on the angle between the camera vector and the normal vector, the second selection method may be based on the shooting resolution, and the third selection method may be based on the position of the area corresponding to the patch. In this case, first, an image is selected using the first selection method, that is, the method based on the angle between the camera vector and the normal vector. If an image cannot be selected using the first selection method, the second selection method is used. If an image cannot be selected using the first selection method, for example, if there are multiple images with the same angle. In this case, an image is selected from among the images with the same angle using the second selection method. 【0104】 Alternatively, the results of each selection method can be comprehensively evaluated to select an image corresponding to the patch. For example, the results of each selection method can be ranked, and a predetermined score can be assigned to each rank. Then, the sum of these scores can be calculated, and the image with the highest total score can be selected as the image corresponding to the patch. In this case, differences in the scores assigned to the results of each selection method may be applied; that is, weighting may be used. 【0105】 [Other embodiments and variations] [About 3D patch models] In the above embodiment, we described an example where a 3D patch model is generated in which each patch is composed of a triangle (triangular mesh). However, the shape of each patch constituting the 3D patch model is not limited to this. For example, it is also possible to generate a model in which each patch is composed of a quadrilateral (quadrilateral mesh). 【0106】 [Regarding filming techniques] The images to be processed only need to be taken from multiple viewpoints with overlapping shooting ranges, and the shooting method is not particularly limited. Images can also be taken using cameras mounted on unmanned aerial vehicles such as drones. 【0107】 Furthermore, the images to be processed include those extracted from video footage. In other words, the subject can be filmed using video footage, and each frame of the filmed video footage can be used as the image to be processed. 【0108】 [Regarding the measurement target] In the above embodiment, the example of measuring the width of cracks appearing on the surface of a bridge pier was used, but the application of the present invention is not limited to this. It can be applied to measuring the size of damage appearing on the surface of structures, especially concrete structures, from images. Structures include bridges, tunnels, dams, buildings, etc. Roads are also included. 【0109】 Furthermore, although the above embodiment was described using the example of measuring the width of a crack, the present invention can also be applied to measuring other types of damage. In particular, it is effective when affected by tilting. Therefore, as long as it is affected by tilting, it can also be applied to measuring the size (area, etc.) of damage such as peeling, exposed rebar, water leakage (including rust stains), free lime, and corrosion. 【0110】 Furthermore, regarding cracks, the present invention can also be applied when measuring their length, especially when the length of the crack is affected by the force of the crack. 【0111】 [Regarding the measurement of crack length] As described above, the present invention can also be applied when measuring the length of cracks. 【0112】 Furthermore, when measuring the length of a crack, the following method can be used: Extract patches where cracks are continuous, and sum the lengths of the cracks obtained in each extracted patch to measure the length of the continuous crack. In other words, measure the length of cracks detected across multiple patches. If the length of the cracks has been corrected, sum the corrected lengths to measure the length of the continuous crack. 【0113】 Figure 15 is a conceptual diagram of a method for measuring the length of a crack. 【0114】 In the figure, patches P1 to P13, indicated by dashed lines, are patches containing a continuous crack C. In this case, the sum of the crack lengths measured in each patch P1 to P13 is considered to be the length of the continuous crack C. That is, if the crack lengths in each patch P1, P2, ..., P13 are denoted as CL1, CL2, ..., CL13, then the length CL of the continuous crack C is calculated as CL = CL1 + CL2 + ... + CL13. 【0115】 Similarly, when measuring the area of damage, the area of damage measured at each patch can be added together to measure the area of damage detected across multiple regions. 【0116】 [Regarding the configuration of the image processing device] The functions of an image processing device are realized by various types of processors. These processors include general-purpose processors such as CPUs and / or GPUs (Graphics Processing Units) that execute programs and function as various processing units; programmable logic devices (PLDs) such as FPGAs (Field Programmable Gate Arrays) whose circuit configurations can be changed after manufacturing; and dedicated electrical circuits such as ASICs (Application Specific Integrated Circuits) that have circuit configurations specifically designed to perform particular processing. A program is synonymous with software. 【0117】 A single processing unit may be composed of one of these various processors, or it may be composed of two or more processors of the same or different type. For example, a single processing unit may be composed of multiple FPGAs, or a combination of a CPU and an FPGA. Alternatively, multiple processing units may be composed of a single processor. Examples of composing multiple processing units with a single processor include, firstly, a configuration where one or more CPUs and software are combined to form a single processor, and this processor functions as multiple processing units, as is typical of computers used as clients or servers. Secondly, a configuration where a processor is used that realizes the functions of the entire system, including multiple processing units, on a single IC (Integrated Circuit) chip, as is typical of a System on Chip (SoC). Thus, various processing units are configured, in terms of hardware structure, using one or more of the above-mentioned various processors. 【0118】 [others] The method for selecting images corresponding to patches, as described in the second embodiment above, including variations, is also effective even when the size of the damage is not corrected. In other words, since images suitable for measuring damage can be selected, damage can be measured with higher accuracy than usual. "Normally" refers to the case where images corresponding to patches are selected without selecting images from the perspective described in the second embodiment above. Furthermore, when generating a 3D shape model, better images can be applied to each patch. This makes it possible to generate a higher-quality 3D shape model. 【0119】 Figure 16 is a flowchart showing the procedure for measuring the size of damage without correction. 【0120】 First, a process is performed to acquire a set of images of the subject (step S31). Next, a process is performed to detect damage from each acquired image (step S32). For example, a process is performed to detect cracks. Next, a process is performed to measure the size of the damage detected in each image (step S33). For example, the width of the cracks is measured. Next, a process is performed to generate 3D point cloud data of the subject from the acquired set of images (step S34). For example, 3D point cloud data of the subject is generated by SfM and MVS. Next, a process is performed to generate a 3D patch model of the subject based on the generated 3D point cloud data of the subject (step S35). For example, a TIN model is generated using 3D Delaunay triangulation. Next, texture mapping is performed on the generated 3D patch model to generate a 3D shape model of the subject (step S36). At this time, images are selected using the method described in the second embodiment (including the modified method). Specifically, images corresponding to patches are selected by one of the following methods, or a combination thereof: (a) selecting an image corresponding to a patch based on the angle between the camera vector and the normal vector; (b) selecting an image based on the shooting resolution; or (c) selecting an image based on the position of the area corresponding to the patch. As a result, an image with a highly accurate measurement of the size of the damage is applied to each patch. With the corrections of the present invention applied, even more accurate measurements become possible. 【0121】 In this example as well, for continuous cracks, it is preferable to sum the lengths of the cracks measured in each patch and measure the total length. 【0122】 [Note] The following additional information is disclosed regarding the embodiments described above. 【0123】 (Note 1) Equipped with a processor, The aforementioned processor, The process involves acquiring multiple images by capturing the subject from multiple viewpoints with overlapping shooting ranges using a camera, and A process for detecting damage to the surface of the subject from multiple acquired images, The process involves analyzing multiple acquired images to generate 3D point cloud data of feature points, A process to generate a 3D patch model of the subject based on the generated point cloud data, Image selection process to select images corresponding to each patch of the 3D patch model from among the acquired multiple images, The process involves measuring the size of the damage in the region corresponding to the patch in the selected image, To do Image processing device. 【0124】 (Note 2) The processor, in the image selection process, selects the image in which the angle θ between the camera vector a and the normal vector b is smallest. The image processing apparatus described in Appendix 1. 【0125】 (Note 3) The processor selects the image with the highest capture resolution in the image selection process. The image processing apparatus described in Appendix 1. 【0126】 (Note 4) In the process of selecting an image corresponding to the patch, the processor selects an image in which the location of the region corresponding to the patch is closest to the center of the image. The image processing apparatus described in Appendix 1. [Explanation of Symbols] 【0127】 1. A system for measuring the width of cracks. 10 Cameras 100 Image Processing Devices 101 CPU 102 RAM 103 ROM 104 Auxiliary storage 105 Communication Interface 106 Input device 107 Display device 111 Image acquisition unit 112 Point Cloud Data Generation Unit 113 3D Patch Model Generation Unit 114 3D Shape Model Generation Unit 114A Image Selection Unit 115 Crack detection section 116 Crack Measurement Section 117 Correction Value Calculation Unit 118 Measurement Result Correction Unit a Camera vector a1 First camera vector a2 Second camera vector b Normal vector θ is the angle between the camera vector and the normal vector. C cracks CL1~CL13 Crack length I1i Image 1 I2i 2nd image IG image collection II Image OL overlap SL Side Wrap Ob Subject (Bridge Pier) P Patch P1-P13 Patches including cracks Pc1 1st position Pc2 2nd position R Shooting range S Surface containing cracks S1-S6 Procedure for SfM Processing S11-S18 Image Processing Procedure S21-S24 Procedure for image selection process S31-S36 Procedure for measuring the size of damage without correction.
Claims
[Claim 1] Equipped with a processor, The aforementioned processor, The camera captures multiple images of the subject from multiple viewpoints, overlapping the shooting ranges. Damage to the surface of the subject is detected from the multiple images obtained. The size of the detected damage is measured, The acquired multiple images are analyzed to estimate the position and orientation at which each image was taken, and to generate three-dimensional point cloud data of feature points. Based on the generated point cloud data, a three-dimensional patch model of the subject is generated. From the multiple images acquired, select the image corresponding to each patch of the three-dimensional patch model. Based on the camera vector a connecting the camera position when the image corresponding to the patch was captured and the patch, and the normal vector b of the patch, a correction value is calculated to correct the measurement result of the size of the damage included in the image corresponding to the patch. Using the calculated correction value, the measurement result of the size of the damage included in the image corresponding to the patch is corrected. If multiple images correspond to the patch, a predetermined score is assigned to each image in order of increasing angle θ between the camera vector a and the normal vector b, decreasing resolution, and decreasing position of the area corresponding to the patch closest to the center of the image. The image with the highest total score is selected as the image corresponding to the patch. Image processing device. [Claim 2] The processor uses the camera vector a as the vector connecting the camera position when the image applied to the patch is captured and the patch, calculates the cosine value cosθ of the angle θ between the camera vector a and the normal vector b of the patch, and calculates the correction value. The image processing apparatus according to claim 1. [Claim 3] The processor calculates the cosine value cosθ using cosθ = a・b / |a||b|. The image processing apparatus according to claim 2. [Claim 4] The processor corrects the measurement result of the size of the damage by dividing it by the cosine value cosθ. The image processing apparatus according to claim 2 or 3. [Claim 5] The processor generates the patches using a triangular mesh or a quadrilateral mesh, and generates the three-dimensional patch model. The image processing apparatus according to any one of claims 1 to 4. [Claim 6] The processor calculates the physical dimensions of the patch in the three-dimensional patch model. The image processing apparatus according to any one of claims 1 to 5. [Claim 7] The processor calculates the physical dimensions of the patch based on the distance measurement information associated with the image. The image processing apparatus according to claim 6. [Claim 8] The processor calculates the physical dimensions of the patch based on the reference point information contained in the image. The image processing apparatus according to claim 6. [Claim 9] The damage is a crack, and the width and / or length of the crack is measured. The image processing apparatus according to any one of claims 1 to 8. [Claim 10] The processor extracts the patches in which the cracks are continuous, and further measures the length of the continuous cracks by summing the corrected crack lengths obtained for each extracted patch. The image processing apparatus according to claim 9. [Claim 11] The camera captures multiple images of the subject from multiple viewpoints, overlapping the shooting ranges. Damage to the surface of the subject is detected from the multiple images obtained. The size of the detected damage is measured, The acquired multiple images are analyzed to estimate the position and orientation at which each image was taken, and to generate three-dimensional point cloud data of feature points. Based on the generated point cloud data, a three-dimensional patch model of the subject is generated. From the multiple images acquired, select the image corresponding to each patch of the three-dimensional patch model. Based on the camera vector a connecting the camera position when the image corresponding to the patch was captured and the patch, and the normal vector b of the patch, a correction value is calculated to correct the measurement result of the size of the damage included in the image corresponding to the patch. Using the calculated correction value, the measurement result of the size of the damage included in the image corresponding to the patch is corrected. If multiple images correspond to the patch, a predetermined score is assigned to each image in order of increasing angle θ between the camera vector a and the normal vector b, decreasing resolution, and decreasing position of the area corresponding to the patch closest to the center of the image. The image with the highest total score is selected as the image corresponding to the patch. Image processing methods. [Claim 12] A function that uses a camera to capture multiple images of a subject from multiple viewpoints with overlapping shooting ranges, A function to detect damage to the surface of the subject from multiple acquired images, A function to measure the size of the detected damage, The system includes a function to analyze multiple acquired images to estimate the position and orientation at which each image was taken, and to generate three-dimensional point cloud data of feature points. A function to generate a three-dimensional patch model of the subject based on the generated point cloud data, A function to select images corresponding to each patch of the 3D patch model from among the multiple images acquired, A function to calculate a correction value for correcting the measurement result of the size of the damage included in the image corresponding to the patch, based on a camera vector a connecting the camera position when the image corresponding to the patch was taken and the patch, and the normal vector b of the patch, A function to correct the measurement result of the size of the damage included in the image corresponding to the patch using the calculated correction value, If multiple images correspond to the aforementioned patch, a predetermined score is assigned to each image in order of increasing angle θ between the camera vector a and the normal vector b, in order of decreasing resolution, and in order of proximity of the area corresponding to the patch to the center of the image. The image with the highest total score is then selected as the image corresponding to the patch. An image processing program that enables computers to perform image processing.