Image processing apparatus, image display apparatus, image processing method, and image processing program

Abstract

To perform efficient inspection while generating point cloud data.SOLUTION: An image processing apparatus (20) includes: an image acquisition unit (21) which acquires captured images obtained by imaging a target while moving along a predetermined path in each of multiple time zones; an image change detection unit (24) which detects a change in the target on the basis of the captured images captured in each time zone; a point cloud generation unit (22) which generates point cloud data representing the target on the basis of the captured images captured in the time zones; and a three-dimensional change detection unit (23) which detects a change in the target on the basis of the point cloud data.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, and an image processing program. [Background technology]

[0002] It is important to check whether any abnormalities have occurred in buildings due to aging or natural disasters in order to prevent serious accidents such as the collapse of the building.

[0003] Conventionally, inspection of buildings as described above has been performed visually by humans, but in recent years, an inspection system has been proposed in which point cloud data is generated from a group of captured images captured by a mobile body (e.g., a drone) equipped with an imaging device, and inspection is performed based on the point cloud data. A known technique for generating point cloud data from captured images in such an inspection system is described in Patent Document 1. Patent Document 1 describes a technique in which a mobile body equipped with a camera generates point cloud data by a structure from motion (SFM) method from captured images of a building captured while moving. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2015-114954 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the technology described in Patent Document 1, in order to realize SFM, it is necessary to increase the overlap between captured images in the previous and next captures, so a large amount of capture is required. Therefore, it is necessary to process a large amount of images to generate point cloud data, which has the problem of taking a lot of time. Therefore, it is not suitable for inspection work that requires frequent execution, and is an obstacle to effective use in business.

[0006] The present invention has been made in consideration of the above problems, and aims to realize a technology that enables efficient inspection while generating point cloud data. [Means for solving the problem]

[0007] In order to solve the above problems, an image processing device according to one embodiment of the present invention is characterized in that it includes an image acquisition unit that acquires images of an object while moving along a predetermined route in each of a plurality of time periods, an image anomaly detection unit that detects abnormality in the object based on the images captured in each time period, a point cloud generation unit that generates point cloud data representing the object based on the images captured in the plurality of time periods, and a 3D abnormality detection unit that detects abnormality in the object based on the point cloud data.

[0008] In order to solve the above problems, an image processing method according to one embodiment of the present invention is characterized in that it executes an image acquisition process for acquiring images of an object while moving along a predetermined route in each of a plurality of time periods, an image anomaly detection process for detecting an abnormality in the object based on the images acquired in each time period, a point cloud generation process for generating point cloud data representing the object based on the images acquired in the plurality of time periods, and a 3D abnormality detection process for detecting an abnormality in the object based on the point cloud data. Effect of the Invention

[0009] According to one aspect of the present invention, efficient inspection can be performed while generating point cloud data. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing a configuration of an image display device according to a first embodiment of the present invention. [Diagram 2] FIG. 1 is a diagram illustrating an imaging device according to a first embodiment of the present invention. [Diagram 3]FIG. 4 is a diagram for explaining an overlap rate between captured images in the first embodiment of the present invention. [Figure 4] 3A to 3C are diagrams showing examples of images captured in respective time periods in the first embodiment of the present invention. [Diagram 5] FIG. 2 is a diagram showing an example of an imaging method for each time period in the first embodiment of the present invention. [Figure 6] FIG. 4 is a diagram showing an overlap rate after integration in the first embodiment of the present invention. [Figure 7] FIG. 4 is a diagram showing another example of an imaging method for each time period in the first embodiment of the present invention. [Figure 8] FIG. 4 is a diagram showing an example of a method for calculating the volume of a deformation in the first embodiment of the present invention. [Figure 9] FIG. 4 is a diagram showing an example of a method for calculating an area of ​​a deformation in the first embodiment of the present invention. [Figure 10] 4 is a flowchart showing the flow of an image processing method implemented by the image processing device according to the first embodiment of the present invention. [Figure 11] FIG. 2 is a diagram showing an example of a display screen of the image display device according to the first embodiment of the present invention. [Figure 12] FIG. 11 is a block diagram showing a configuration of an image display device according to a second embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] [Embodiment 1] Hereinafter, one embodiment of the present invention will be described in detail with reference to FIGS.

[0012] (Overview of image display device 10) An image display device 10 according to an embodiment of the present invention is a device that includes an image processing device that detects an abnormality of an object X based on captured images of the object X captured while moving along a predetermined route or on point cloud data generated from the captured images. The image display device 10 is also a device that obtains an abnormality detection result from the image processing device 20 and displays the detection result to a user.

[0013] In this specification, the term "target" refers to a target to be inspected during a regular inspection or an extraordinary inspection, such as a structure or a natural object. More specifically, examples of the target include a reservoir slope of a dam, a building, a road, a bridge, or a natural object such as the sea, a river, or a mountain.

[0014] (Configuration of image display device 10) The schematic configuration of an image display device 10 will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the configuration of an image display device.

[0015] As shown in FIG. 1, the image display device 10 includes an image processing device 20, a storage unit 12, and a display unit 13.

[0016] The image display device 10 is realized as various devices capable of displaying images. Devices that can be used as the image display device 10 include a monitor, a smartphone, a PC (Personal Computer), a tablet PC, and the like.

[0017] [Image processing device 20] The image processing device 20 includes an image acquisition unit 21, a point cloud generation unit 22, a 3D deformation detection unit 23, and an image deformation detection unit 24, and detects deformation of the object X based on the acquired captured images, generates point cloud data from the acquired captured images, and detects deformation of the object X based on the point cloud data. The image processing device 20 also outputs the deformation detection result to the storage unit 12 or the display unit 13. The image processing device 20 is composed of, for example, a CPU (Central Processing Unit) or a GPU (Graphic Processing Unit: image processing device), etc.

[0018] (Image acquisition unit 21) The image acquisition unit 21 acquires captured images of the object X while moving along a predetermined route in each of a plurality of time periods. As an example, the image acquisition unit 21 acquires captured images of the object X while the imaging device 30 moves along a predetermined route. In this embodiment, a moving body (e.g., a drone) equipped with a camera is adopted as the imaging device 30. The moving body captures the object X while moving along a predetermined route several times a day at different time periods, and stores the captured captured images in a storage device provided in the main body or transmits them to an external computer. The image acquisition unit 21 acquires captured images of the object X at different time periods by reading the captured images stored in the storage device of the moving body or the captured images received by an external computer. Note that the predetermined route here is a route that can be set arbitrarily, for example, a route that can capture the entire object X in the shortest time.

[0019] Hereinafter, an imaging method in which the imaging device 3 captures an image of the object X will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of an imaging method in which the imaging device 30 captures an image of the object X. In Fig. 2, only the camera 301 included in the imaging device 30 is illustrated and described.

[0020] 2, camera 301 is installed in imaging device 30 so that its lens faces vertically downward, and captures an image of object X from directly above while moving in the direction of the arrow. The orientation of camera 301 can be adjusted as desired. For example, camera 301 may be installed in imaging device 30 so that its lens faces diagonally, and image object X from a diagonal direction. Camera 301 may also be installed in imaging device 30 so that its lens faces vertically horizontally, and image the side of object X, or camera 301 may be installed in imaging device 30 so that its lens faces vertically upward, and image the back of object X.

[0021] (Point cloud generation unit 22) The point cloud generating unit 22 generates point cloud data representing the object X based on the captured images captured in a plurality of time periods acquired by the image acquiring unit 21. As an example, the point cloud generating unit 22 generates the point cloud data by using a known technique such as the SFM method. The SFM method is a method of extracting corresponding points in overlapping areas between captured images, estimating the position and orientation of the camera that captured the images from the corresponding points, and outputting three-dimensional information of the corresponding points. The three-dimensional information output here is point cloud data. To generate point cloud data of an object using the SFM method, captured images in which at least a portion of the imaging range overlaps are required. In addition, captured images in which the overlap rate, which indicates the ratio of the overlapping areas between the captured images, is equal to or greater than a predetermined value are required. Therefore, the point cloud generating unit 22 acquires captured images in which a portion of the imaging range overlaps from the image acquiring unit 21, and generates point cloud data of the object X using the SFM method.

[0022] Hereinafter, the overlap rate between captured images will be described with reference to FIG. 3. FIG. 3 is a diagram showing a state in which a moving body equipped with a camera 301 captures an image of an object X while moving. As shown in FIG. 3, the moving body equipped with a camera 301 captures an image of the object X while moving in the direction of a dashed arrow. As an example, the camera 301 captures an image of the object X n times. In FIG. 3, the imaging ranges of the camera 301 at the i-th time (i is a natural number between 1 and n) and the i+1th time are indicated by solid triangles. The overlap rate between the captured image captured at the i-th time and the captured image captured at the i+1th time is calculated using the following formula (1) from the ground distance X, which is the ground distance of the imaging range, and the imaging baseline length Y, which is the center distance of the camera 301 in each of the n-th time and the n+1th time imaging.

[0023]

number

[0024] Here, the overlap rate is R, the ground distance X is S, and the shooting baseline length Y is B. Equation (1) calculates the overlap rate R as the proportion of the overlapping area Z between the i-th imaging area and the i+1-th imaging area in the ground distance X. If the i-th imaging and the i+1-th imaging are performed in exactly the same position, the overlap rate R is 100%, and if the i-th imaging and the i+1-th imaging are performed in a position where the shooting baseline length Y is greater than the ground distance X, the overlap rate R is 0%.

[0025] The overlap rate required for generating point cloud data varies depending on the algorithm of the SFM method used to generate the point cloud data and the number of feature points of the subject, so the overlap rate R can be set to any value, such as 70% or 90%. That is, in generating point cloud data in the conventional technology, images of the subject are captured multiple times while moving, and the images are captured so that the overlap rate R between the imaging areas of the previous and following times (for example, the imaging area of ​​the i-th time and the imaging area of ​​the i+1th time) is the overlap rate required for generating point cloud data. This makes it possible to generate point cloud data from the obtained images.

[0026] In contrast, in this embodiment, the moving body captures the object X while moving along a predetermined route several times a day at different times. The moving body captures the object X while shifting the imaging position for each time period. In this case, the overlap rate R of the imaging areas before and after imaging in a certain time period does not need to be the overlap rate required for generating point cloud data, and the point cloud data can be generated as long as the overlap rate R' obtained by integrating the image groups obtained by imaging in each time period is the overlap rate required for generating point cloud data. This will be described in detail below.

[0027] First, an example of captured images in each time period will be described with reference to Fig. 4. Fig. 4 is a diagram showing the correspondence relationship between captured images captured in each time period. In Fig. 4, a solid-line rectangle represents an imaging range captured in one imaging session.

[0028] For example, in the imaging at time N, the moving body moves along a predetermined route and images the target X such that the overlap rate R of each captured image is 10%. This allows the predetermined route to be imaged without interruption in the imaging at time N. In the imaging at time N+1, the moving body also moves along a predetermined route and images the target X such that the overlap rate R of each captured image is 10%. This allows the predetermined route to be imaged without interruption in the imaging at time N+1. Then, by taking the difference between the captured image at time N and the captured image at time N+1, it is possible to detect a mutation as described below. Note that the overlap rate R of each captured image is not limited to 10%, but may be 0% or more, and may be appropriately set according to the accuracy of the imaging position so as not to miss any images.

[0029] Furthermore, the moving body captures images such that the imaging position of each captured image at time N+1 is shifted from the imaging position of each captured image at time N by 10% of the width of the captured image in the traveling direction. As a result, the captured image at time N and the captured image at time N+1 overlap by 90%. By similarly shifting the imaging position for imaging from time N+2 onwards, when the captured images from N to N+9 are integrated, an image similar to that obtained when images are captured in one time period with an overlap rate R of 90% is obtained. This is expressed as the overlap rate R' of the integrated captured images being 90%, and if the overlap rate required for generating point cloud data is 90%, point cloud data can be generated from the integrated captured images.

[0030] In this manner, if the overlap rate R of the captured images captured in each time period is preset to 10%, and the overlap rate required for generating point cloud data is preset to 90%, when the point cloud generating unit 22 acquires the captured images in nine time periods from the image acquiring unit 21, it may determine that the overlap rate R' of the integrated captured images is equal to or greater than the overlap rate required for generating point cloud data, and generate point cloud data representing the object X. Alternatively, the overlap rate R' may be calculated based on the correspondence between the integrated captured images, and if it is equal to or greater than the overlap rate required for generating point cloud data, the point cloud generating unit 22 may generate point cloud data representing the object X.

[0031] A specific example of a method for calculating the overlap rate R' based on the correspondence between the integrated captured images will be described below. First, the amount of shift in the image capturing position between the captured images captured in each time period is calculated using the timing of capturing images in the capturing path of each captured image to be integrated, GPS (Global Positioning System) coordinates, etc. The overlap rate R' when the captured images captured in each time period are integrated based on the amount of shift is calculated. If the calculated amount of shift is equal to or less than a predetermined value, the image pair may be determined to have similar image capturing positions and capturing directions of the object X, and the overlap rate R' may be calculated based on the difference in pixel values ​​between the image pair. Specifically, the image pair with the smallest pixel value difference may be specified, and the overlap rate R' may be calculated from the amount of shift in the image capturing positions of the image pair. If the overlap rate R' in all captured images satisfies the overlap rate required for generating point cloud data when each image pair of the integrated captured images is integrated, it may be determined that the overlap rate is equal to or greater than the overlap rate required for generating point cloud data, and point cloud data representing the object X may be generated.

[0032] Here, the predetermined value of the amount of deviation used as a criterion for determining that the image pair has similar imaging positions and imaging directions of the object X can be set arbitrarily, and as an example, it is the value of the ground distance X. This makes it possible to extract captured images in which at least a part of the imaging area overlaps, and to make them into an image pair.

[0033] If the captured images are blurred, shaken, overexposed, or underexposed and it is difficult to associate the captured images, it is determined that the overlap rate required for generating point cloud data has not been reached.

[0034] An example of an imaging method in each time period will be described below with reference to Fig. 5. Fig. 5 is a diagram showing an imaging method in each time period. In Fig. 5, the imaging positions where the camera 301 equipped on the moving object images the object X are indicated by symbols. Also, the imaging range when imaging at each imaging position is indicated by a solid-line triangle.

[0035] For example, in imaging at time N, the camera 301 images the object X at imaging positions 301A1, 301A2, and 301A3. Here, the overlap rate R is 20%. In imaging at time N+1, the camera 301 images the object X at imaging positions 301B1, 301B2, and 301B3. Here, the imaging position 301B1 is a position shifted by the overlapping area Z between the imaging positions 301A1 and 301A2 from the imaging position 301A1 at the time of imaging at N, that is, 80% of the imaging range imaged from the imaging position 301A1 and the imaging range imaged from the imaging position 301B1 overlap. Similarly, in times N+2 and N+3, the camera 301 images the object X while moving, shifting it by the overlapping area Z from the previous time slot, so that the overlap rate R is 20%.

[0036] Fig. 6 shows the imaging range when images captured every hour from N o'clock to N+3 o'clock are integrated. As shown in Fig. 6, when the images captured from N o'clock to N+3 o'clock are integrated, the ratio of the overlapping area Z' to the ground distance X of each imaging range in each time period is 80%. In other words, the overlap rate R' of the integrated captured images is 80%. If the overlap rate R' satisfies the overlap rate required for generating point cloud data, the point cloud generation unit 22 generates point cloud data representing the target X from the acquired integrated captured images with an overlap rate R' of 80%.

[0037] As described above, the overlap rate R of the captured images taken in each time period is small (sparse), but by using a method of capturing images of the object X while shifting the capturing position, it is possible to efficiently capture images of the object X in each time period and obtain captured images required for generating point cloud data.

[0038] In this way, the captured images taken in each time period include at least an image of the object X captured from a different position from the captured images captured in other time periods, so that the captured images taken in a certain time period include images captured from a different position from the captured images captured in other time periods, and thus by integrating the captured images taken in a plurality of time periods, an image with a large overlap rate R' can be obtained. This allows the point cloud data to be successfully generated.

[0039] In this embodiment, the method of capturing an image of the object X in each time period is, but not limited to, a method of capturing an image while shifting the image capturing position toward the moving direction of the moving object. For example, as shown in FIG. 7, the image capturing may be performed while changing the direction of shifting the image capturing position so that the difference in image capturing time between adjacent captured images is reduced when the captured images are integrated. By capturing an image while changing the direction of shifting the image capturing position for each time period, the difference in image capturing time between adjacent captured images in the integrated captured image is reduced. The smaller the difference in image capturing time, the smaller the change in the sunlight conditions and the object X, and the better the correspondence between the captured images can be performed.

[0040] Furthermore, in this embodiment, the object X is imaged every hour from N o'clock to N+3 o'clock, but it is sufficient to image the object X in a plurality of time periods, and the image capturing intervals may be indefinite.

[0041] (3D Deformation Detection Unit 23) The three-dimensional deformation detection unit 23 detects deformation of the object X based on the point cloud data (hereinafter referred to as "generated point cloud data") generated by the point cloud generation unit 22. The three-dimensional deformation detection unit 23 detects deformation of the object X, for example, by comparing the generated point cloud data with comparison point cloud data. Here, the comparison point cloud data may be point cloud data generated by the point cloud generation unit 22 in the past. For example, when point cloud data is generated every day based on captured images captured at different times of day several times, the point cloud data generated one day before may be used as the comparison point cloud data.

[0042] As an example, the 3D deformation detection unit 23 detects deformation of the object X by comparing the generated point cloud data with the comparison point cloud data and determining the difference. The deformation detected here is a change in volume, such as a slope collapsing to form a dent and a decrease in volume. The 3D deformation detection unit 23 takes the difference between the generated point cloud data and the comparison point cloud data, and performs mesh processing on the point cloud of the difference to calculate the volume. When the volume is equal to or greater than a predetermined value, the unit outputs information on the volume of the deformation as information indicating the position of the deformation and the degree of the deformation to the storage unit 12 and the display unit 13 as there is a deformation. When the volume is smaller than the predetermined value, the unit outputs information indicating there is no deformation to the storage unit 12 and the display unit 13 as there is no deformation.

[0043] An example of a method for calculating the volume of a deformation from the generated point cloud data and the comparison point cloud data will be described below with reference to Fig. 8. Fig. 8 is a diagram showing the procedure for calculating the volume of a detected deformation. In Fig. 8, the generated point cloud data is indicated by black circles, and the comparison point cloud data is indicated by white circles. Note that Fig. 8 shows cross sections at any depth of the generated point cloud data and the comparison point cloud data in two dimensions.

[0044] First, as shown in Fig. 8, the 3D deformation detection unit 23 overlaps point clouds that represent the same position of the target X in the generated point cloud data and the comparison point cloud data (see Fig. 8(1)) (see Fig. 8(2)). Next, the overlapping point clouds are deleted, and only non-overlapping point clouds are left, thereby obtaining the difference between the generated point cloud data and the comparison point cloud data (see Fig. 8(3)). Next, a meshing process is performed on the point clouds that represent the difference among the generated point cloud data and the comparison point cloud data, to calculate the volume V (see Fig. 8(4)).

[0045] In this embodiment, the point cloud data of the object X generated one day before is used as the comparison point cloud data, but the present invention is not limited to this. For example, the comparison point cloud data may be point cloud data generated from a three-dimensional drawing of the object X or point cloud data generated using a measuring device such as a laser scanner.

[0046] In addition, in this embodiment, the 3D deformation detection unit 23 compares the generated point cloud data with the comparison point cloud data to detect the deformation of the object X, but the present invention is not limited to this. For example, a deformation detection range may be specified as the target for detecting the deformation. More specifically, the deformation detection range is limited to a location where the deformation is particularly desired to be detected, such as the reservoir slope of a dam, and the deformation is detected. This reduces false detections due to changes in volume caused by the presence or absence of cars when there are roads or parking lots around the reservoir. Furthermore, the time required for the deformation detection can be shortened by limiting the range for the 3D deformation detection.

[0047] (Image Deformation Detection Unit 24) The image abnormality detection unit 24 detects an abnormality of the object X based on the captured images captured in each time period. For example, the image acquisition unit 21 acquires captured images of the object X in different time periods, and compares the captured images captured in the predetermined time period with the captured images captured in a time period different from the predetermined time period to detect the abnormality of the object X. Here, the predetermined time period is any of the multiple time periods in which the captured images acquired from the image acquisition unit 21 were captured. In addition, the time period different from the predetermined time period is, for example, an image captured of the object X before the predetermined time period. More specifically, it is an image captured in the time period immediately before the predetermined time period, an image captured in the same time period in the past, an image captured when the object X was visually confirmed to be normal by a person, and the like. As an example, the image abnormality detection unit 24 compares the captured images captured in the predetermined time period with the captured images captured in a time period different from the predetermined time period, and obtains a difference. When the area of ​​the difference is equal to or greater than a predetermined value, the area of ​​the deformation is determined to be deformed, and information on the area of ​​the deformation as information showing the degree of the deformation is output to the storage unit 12 and the display unit 13. When the area of ​​the difference is smaller than the predetermined value, the area of ​​the difference is determined to be deformed, and information to that effect is output to the storage unit 12 and the display unit 13.

[0048] Hereinafter, a specific example of a method in which the image anomaly detection unit 24 detects anomalies will be described with reference to Fig. 9. Fig. 9 is a diagram for explaining a specific example of a method for detecting anomalies from images captured in different time periods. Fig. 9 shows an example in which the predetermined time period is N o'clock, and a time period different from the predetermined time period is N-1 o'clock, which is one time period before the predetermined time period.

[0049] In FIG. 9, the difference between a captured image 1201 captured at time N and a captured image 1202 captured at time N-1 is taken. In order to integrate captured images from a plurality of time periods and obtain a captured image with a large overlap rate R', the captured images 1201 at time N and the captured image 1202 at time N-1 are positioned differently from each other, so the captured images are aligned before taking the difference between the captured images. The alignment between the captured images is performed, for example, by obtaining a pair of two corresponding pixels between the captured images, that is, corresponding points, and calculating a geometric transformation parameter of the image based on the corresponding points to obtain a correspondence relationship between all pixels between the captured images. After aligning the captured images 1201 at time N and 1202 at time N-1, a difference image 1203 is generated, which is an image in which the magnitude of the difference is expressed by a luminance value, and the area of ​​a region with a large difference is obtained from the difference image 1203. In the difference image 1203, a large luminance value, that is, a white portion, indicates a large difference. In Fig. 9, when the captured image at time N and the captured image at time N-1 are aligned and the difference is taken, the difference is large because the water surface area 1204 in the captured image 1201 at time N has expanded, and appears as a white area in the difference image 1203. The area of ​​this area 1205 is calculated, and if the area is equal to or greater than a predetermined value, information regarding the area and position of the abnormality is output as an abnormality. Also, if the area of ​​area 1205 is smaller than the predetermined value, information indicating that no abnormality has occurred is output.

[0050] As described above, it is preferable that the captured images taken in each time period are images captured such that images captured before and after the time period overlap each other. This allows captured images of the target to be obtained continuously for each time period, so that the image anomaly detection unit 24 can successfully detect anomalies based on the captured images.

[0051] As described above, in this embodiment, even if the overlap rate required for generating point cloud data is not reached, the image processing device 20 can detect the deformation of the object X by the image deformation detection unit 24 comparing the captured images of each time period. In other words, the image processing device 20 generates point cloud data for each day and compares them to detect the deformation of the object X, so that it is possible to detect daily changes, and also to detect the changes in the object X for each time period by comparing different time periods in a day, so that more detailed inspection is possible. In this way, according to the image processing device 20, it is possible to perform efficient inspection based on the captured images captured in each time period while generating point cloud data based on the captured images captured in a plurality of time periods.

[0052] [Storage section 12] The storage unit 12 stores various information. As an example, the storage unit 12 records images acquired by the image acquisition unit 21 and abnormality detection results output from the image processing device 20 on a recording medium. The storage unit 12 also reads the images and detection results recorded on the recording medium and outputs them to the image processing device 20 and the display unit 13. Any general-purpose storage unit can be used as the storage unit 12 as long as it has the above-mentioned functions.

[0053] [Display section 13] The display unit 13 displays various images. As an example, the display unit 13 displays the abnormality detection results output from the image processing device 20 and images recorded in the storage unit 12. The display unit 13 may be configured to include a display device such as a liquid crystal display (LCD) or an organic electro-luminescence display (OELD).

[0054] (Flow of image processing method M1) In the image display device 10 according to one embodiment of the present invention, the flow of the image processing method M1 implemented by the image processing device 20 in particular will be described with reference to Fig. 10. Fig. 10 is a flowchart showing the flow of each step of the image processing method implemented by the image processing device 20.

[0055] Step S201: The point cloud generating unit 22 and the image abnormality detecting unit 24 of the image processing device 20 acquire from the image acquiring unit 21 captured images of the object X while the moving body moves along a predetermined route in each time period.

[0056] Step S202: The point cloud generating unit 22 integrates the captured images acquired in step S201, and determines whether the overlap rate R' of the integrated captured images is equal to or greater than a predetermined value. The method for calculating the overlap rate R' of the integrated captured images may be a method of calculating based on the overlap rate R of each time period and the shift amount of the imaging position as described above, or a method of calculating based on the correspondence relationship between the captured images acquired in step S201.

[0057] Here, the specified value of the overlap rate R' may be a value input by the user into an input unit (not shown), or it may be a value indicating the overlap rate required to generate point cloud data that is set based on the algorithm of the SFM method used by the point cloud generation unit 22 to generate the point cloud data and the number of feature points of the subject.

[0058] Step S203: If it is determined that the overlap rate R' of the captured images acquired in step S202 is equal to or greater than a predetermined value (YES in step S202), the point cloud generating unit 22 generates generated point cloud data representing the target X from the captured images.

[0059] If it is determined in step S202 that the overlap rate R' of the integrated captured images is smaller than a predetermined value (NO in step S202), the process returns to step S201 and repeats the acquisition of captured images until the overlap rate R' of the integrated captured images satisfies the overlap rate required to generate point cloud data.

[0060] Step S204: The 3D deformation detection unit 23 detects deformation of the object X by comparing the generated point cloud data generated in step S203 with the comparison point cloud data. More specifically, the difference between the generated point cloud data and the comparison point cloud data is calculated, and the difference point cloud is meshed to calculate the volume. If this volume is equal to or greater than a predetermined value, information on the position and volume of the deformation is obtained as a deformation. The information on the position of the deformation is expressed by latitude and longitude and plane rectangular coordinates on each geodetic system. Furthermore, if the volume is smaller than a predetermined value, information on the absence of deformation is obtained.

[0061] Here, the predetermined value of the volume may be a value input by the user to an input unit (not shown), or may be a value set based on a volume value calculated in the past.

[0062] Step S205: The image abnormality detection unit 24 calculates the difference between the image captured in a specific time period and the image captured in a time period different from the specific time period among the captured images acquired in step S201, and if the area of ​​the difference is equal to or greater than a specific value, it determines that there is an abnormality and acquires information on the position and area of ​​the abnormality. The information on the position of the abnormality represents the position of the detected abnormality in latitude and longitude and plane rectangular coordinates on each geodetic system. Also, if the area is smaller than a specific value, it acquires information indicating that there is no abnormality.

[0063] Here, the predetermined value of the area may be a value input by the user to an input unit (not shown), or may be a value set based on a previously calculated area value.

[0064] Step S206: The three-dimensional deformation detection unit 23 outputs the various information acquired in step S204 to the display unit 13 of the image display device 10. For example, if the three-dimensional deformation detection unit 23 detects a deformation, it outputs information about the position and volume of the deformation. If the three-dimensional deformation detection unit 23 detects no deformation, it outputs information indicating that no deformation was detected.

[0065] Step S206: Furthermore, the image abnormality detection unit 24 outputs the various information acquired in step S205 to the display unit 13 of the image display device 10. For example, if the image abnormality detection unit 24 detects an abnormality, it outputs information regarding the position and area of ​​the abnormality. If the image abnormality detection unit 24 detects no abnormality, it outputs information indicating that no abnormality was detected.

[0066] A specific example of a display screen displayed on the display unit 13 will be described with reference to Fig. 11. Fig. 11 is a diagram showing an example of a display screen displayed on the display unit 13 of the image display device 10.

[0067] As shown in FIG. 11, the display screen of the display unit 13 displays an image on an image of the object X captured from above, in which the object X is shown with a solid-line rectangle based on information on the position of the deformation detected in step S204, and information on the volume of the deformation is displayed as text data. This image may also display a dotted-line rectangle representing the position of a past deformation. This dotted-line rectangle can be displayed by reading the past detection results recorded in the storage unit 12, i.e., information on the position and volume of the deformation. This allows the user to compare a deformation detected in the past with a newly detected deformation. For example, if a dotted-line rectangle representing a deformation detected in the past and a solid-line rectangle representing a newly detected deformation overlap, it can be inferred that the deformation has progressed further.

[0068] The timing at which the image processing device 20 executes each process shown in Fig. 10 can be set arbitrarily. For example, the image processing device 20 may execute each time it receives an input of a captured image, or may execute the process every predetermined time (for example, every 30 minutes or every hour).

[0069] (Variation 1) In the above example, the point cloud generating unit 22 uses the captured images acquired by the image acquiring unit 21 to generate generated point cloud data. At this time, if the captured images captured in each time period include a captured image that is not suitable for generating point cloud data, the image acquiring unit 21 may supplement the captured images with another captured image corresponding to the captured image. As an example, the image acquiring unit 21 determines whether the captured image acquired is suitable for generating generated point cloud data. A captured image that is determined to be not suitable for generating generated point cloud data is replaced with a corresponding captured image captured in the same time period in the past.

[0070] Here, the captured image that is not suitable for generating the generated point cloud data is blurred, blurred, weather change, etc. For blurred or blurred images, edges are detected from the captured image, and if the edge amount is equal to or less than a predetermined value, the captured image is determined to be blurred or blurred. For weather change, for example, a captured image in which it is raining or snowing only during a certain time period, and the weather conditions are different from those in other capture time periods. The weather may be estimated from the captured image, or information from a meteorological observation instrument such as a rain gauge installed on a moving body or object X may be input to the point cloud generating unit 22 by an input unit (not shown), and when the value of the rain gauge is equal to or greater than a threshold value, the captured image may be replaced with a captured image captured during the same time period in the past. This makes it possible to ensure the overlap rate required for generating generated point cloud data even when there is a captured image that is not suitable for generating point cloud data. This makes it possible to generate highly accurate point cloud data while ensuring the overlap rate required for generating point cloud data, even if there is a captured image that is not suitable for generating point cloud data.

[0071] Furthermore, when the captured images captured in each time period include an image not suitable for generating generated point cloud data, the image acquisition unit 21 may output a re-imaging instruction to the imaging device 30. For example, when capturing images 12 times a day, capturing images while shifting the position so as to secure an arbitrary overlap rate for nine times, and capturing images in the same position as the time period for which the re-imaging instruction was issued for the remaining three times, it is possible to secure the overlap rate required for generating generated point cloud data in the imaging on the same day.

[0072] (Variation 2) In the above example, the point cloud generating unit 22 generates the generated point cloud data from the captured image input acquired from the image acquiring unit 21, but it is also possible to detect captured images that include subjects that may move in the captured images and exclude them from the generation of the generated point cloud data. Here, subjects that may move include vehicles, people, heavy machinery, etc. This makes it possible to generate the generated point cloud data excluding subjects that may be erroneously detected as abnormalities.

[0073] (Variation 3) In the above example, the 3D deformation detection unit 23 compares the comparison point cloud data generated in advance with the generated point cloud data generated from the captured images acquired by the image acquisition unit 21 to detect and output the 3D deformation, but the detected difference may be saved and an indication of whether the difference is expanding may be output. This makes it possible to check the time series changes in the 3D shape, such as the progress of deformation of the reservoir slope.

[0074] (Variation 4) In the above example, the threshold value of the volume to be detected as a deformation was set before the image processing device 20 performed each step of the image processing method, but the threshold value of the volume may be changed after the 3D deformation detection unit 23 outputs the volume and position of the deformation. This allows the user to set the threshold value of the volume to be detected as a deformation while checking the position and volume of the deformation displayed on the display unit 13, and realizes a display that is easy for the user to see, such as displaying only positions with greater degrees of deformation.

[0075] (Variation 5) In the above example, the image deformation detection unit 24 outputs the calculated area and position of the deformation to the display unit 13, but it may also output them to the point cloud generation unit 22. This allows the point cloud generation unit 22 to generate generated point cloud data from the captured images including the deformation area including the detected deformation, rather than from the entire integrated captured images. Therefore, since point cloud data representing the area including the deformation is generated, the necessary point cloud data can be generated efficiently.

[0076] Furthermore, when the image anomaly detection unit 24 detects an anomaly based on an image captured in a certain time period, the point cloud generation unit 22 may generate point cloud data based on images captured in a plurality of time periods after the certain time period. Since the point cloud data can be generated based on the captured images after the anomaly has occurred, the point cloud data indicating the anomaly can be successfully generated.

[0077] (Variation 6) In the above example, the threshold value of the area to be detected as a deformation was set before the image processing device 20 acquired the captured image from the image acquisition unit 21, but it may be changed after the image deformation detection unit 24 outputs the area and position of the deformation. This allows the user to set the threshold value of the area to be detected as a deformation while checking the position and area of ​​the deformation displayed on the display unit 13, and realizes a display that is easy for the user to see, such as displaying only positions with greater degrees of deformation.

[0078] [Embodiment 2] Another embodiment (embodiment 2) of the present invention will be described in detail with reference to Fig. 12. For ease of explanation, members having the same functions as those described in the above embodiment will be denoted by the same reference numerals, and the description thereof will not be repeated.

[0079] The image display device 10A according to the second embodiment of the present invention differs from the image display device 10A according to the first embodiment in that the image processing device 20 includes an image generating section 25.

[0080] (Image Generation Unit 25) The image generating unit 25 generates an ortho-image based on the point cloud data generated by the point cloud generating unit 22 and the captured image acquired by the image acquiring unit 21. The ortho-image is an image generated by converting the three-dimensional information of the object X obtained from the point cloud data into a directly below viewpoint. At this time, the image generating unit 25 generates the pixel value of each pixel of the ortho-image based on the pixel value of the corresponding pixel of the captured image.

[0081] Here, the image generating unit 25 generates each pixel of the orthoimage based on the pixels of the captured image captured in a specific time period among a plurality of time periods. This makes it possible to successfully obtain an orthoimage with a constant luminance distribution and color even if the luminance distribution and color of the captured image differ depending on the time period, such as an image captured in the morning and an image captured in the evening.

[0082] Furthermore, the image generating unit 25 may configure the orthoimage from a plurality of partial orthoimages, and set the above-mentioned specific time period for each partial orthoimage. By using pixels of different time periods for the area where the change is to be confirmed by the orthoimage (for example, the water surface or the reservoir slope) and other areas, an orthoimage in which the change is easily seen can be obtained. For example, an orthoimage is generated using pixels captured in a predetermined time period for areas other than the reservoir slope, and multiple reservoir slopes are generated for each time period, and displayed in chronological order, so that the change in the reservoir slope in the time axis direction can be easily seen. In this way, an orthoimage in which the change is easily seen can be generated by dividing the orthoimage and generating divided orthoimages using pixels based on images captured in different time periods. The orthoimage generated by the image generating unit 25 may be displayed by the display unit 13.

[0083] [Software implementation example] The control block of the image display device 10 (particularly, each unit in the image processing device 20) may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or may be realized by software.

[0084] In the latter case, the image display device 10 includes a computer that executes instructions of a program, which is software that realizes each function. This computer includes, for example, at least one processor (control device) and at least one computer-readable recording medium that stores the program. The computer also includes an OS (Operating System) and hardware such as peripheral devices. In the computer, the processor reads the program from the recording medium and executes it, thereby achieving the object of the present invention.

[0085] The processor may be, for example, a CPU. The recording medium may be a "non-transitory tangible medium," such as a portable medium such as a ROM (Read Only Memory) or a CD-ROM, a disk such as a tape, a flexible disk or a magneto-optical disk, a card, a semiconductor memory, a programmable logic circuit, or a hard disk built into a computer system. Examples of the computer-readable recording medium include those that dynamically hold a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, and those that can hold a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client.

[0086] The image display device 10 may further include a RAM (Random Access Memory) for expanding the program. The program may be supplied to the computer via any transmission medium capable of transmitting the program (such as a communication network or broadcast waves). One aspect of the present invention may also be realized in the form of a data signal embedded in a carrier wave, in which the program is electronically transmitted and embodied.

[0087] Also, a part or all of the image display device 10 may be realized as an LSI, which is typically an integrated circuit. In this case, the control block of the image display device 10 may be individually chipped, or a part or all of it may be integrated and chipped. The integrated circuit method is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. Furthermore, if an integrated circuit technology that can replace LSI appears due to advances in semiconductor technology, it is also possible to use an integrated circuit based on that technology.

[0088] In addition, in the above-described embodiments, the control lines and information lines indicate those that are considered necessary for explanation, and not all control lines and information lines are necessarily shown in the product, and all components may be interconnected.

[0089] 〔summary〕 The image processing device according to aspect 1 of the present invention comprises an image acquisition unit that acquires images of an object while moving along a predetermined route in each of a plurality of time periods, an image anomaly detection unit that detects abnormality in the object based on the images captured in each time period, a point cloud generation unit that generates point cloud data representing the object based on the images captured in the plurality of time periods, and a 3D abnormality detection unit that detects abnormality in the object based on the point cloud data.

[0090] According to the above configuration, point cloud data can be generated based on the images captured in a plurality of time periods, while efficient inspection can be performed based on the images captured in each time period.

[0091] An image processing device according to aspect 2 of the present invention may be such that, in aspect 1 above, the captured images captured in each time period include at least captured images of the target from a position different from the captured images captured in time periods other than the relevant time period.

[0092] According to the above configuration, since the captured images taken in a certain time period include images captured from different positions from the captured images taken in other time periods, it is possible to obtain captured images with a high overlap rate by integrating the captured images taken in a plurality of time periods, thereby successfully generating point cloud data.

[0093] An image processing device according to aspect 3 of the present invention may be such that, in aspect 2 above, the captured images captured in each time period are comprised of captured images captured such that portions of captured images captured before and after each time period overlap.

[0094] According to the above configuration, it is possible to obtain captured images of an object in succession for each time period, so that abnormalities based on the images can be successfully detected.

[0095] An image processing device according to aspect 4 of the present invention may be configured such that, in aspect 1 above, the image acquisition unit supplements the captured images captured in each time period with another captured image corresponding to the captured image if the captured images captured in each time period include an captured image that is not suitable for generating the point cloud data.

[0096] According to the above configuration, even if there are captured images that are not suitable for generating point cloud data, it is possible to generate highly accurate point cloud data while ensuring an overlap rate necessary for generating point cloud data.

[0097] An image processing device according to aspect 5 of the present invention, in aspect 1 above, may be configured so that, when the image anomaly detection unit detects an anomaly based on an image captured during a certain time period, the point cloud generation unit generates the point cloud data based on the images captured during multiple time periods after the certain time period.

[0098] According to the above configuration, since point cloud data can be generated based on images captured after the deformation has occurred, it is possible to successfully generate point cloud data indicating the deformation.

[0099] An image processing device according to aspect 6 of the present invention may be configured such that, in aspect 1 above, the point cloud generation unit generates point cloud data representing a deformed area including a deformed state when the image deformed state detection unit detects the deformed state.

[0100] According to the above configuration, since point cloud data representing an area including a deformation is generated, necessary point cloud data can be generated efficiently.

[0101] The image processing device according to a seventh aspect of the present invention is the image processing device according to the first aspect, further comprising an image generating unit configured to generate an orthoimage of the object based on the point cloud data and the captured image, The image generating unit may generate each pixel of the orthoimage based on pixels of the captured image captured in a specific time period among the plurality of time periods.

[0102] According to the above configuration, even if the luminance distribution and color tone in the captured image vary depending on the time of day, an orthoimage with a constant luminance distribution and color tone can be successfully acquired.

[0103] An image processing device according to aspect 8 of the present invention, in aspect 7 above, may be such that the image generation unit constructs the orthoimage from a plurality of partial orthoimages, and sets the specific time period for each partial orthoimage.

[0104] According to the above configuration, by dividing an orthoimage and generating a divided orthoimage using pixels based on images captured at different times, it is possible to generate an orthoimage in which changes are easily noticeable.

[0105] An image processing device according to a ninth aspect of the present invention may be configured in such a way that in the first aspect, the three-dimensional deformation detection unit outputs information indicating the position and degree of the detected deformation to a display device.

[0106] According to the above configuration, the user can refer to the abnormality detection results and take necessary measures.

[0107] An image display device according to a tenth aspect of the present invention may include the image processing device according to the ninth aspect.

[0108] According to the above configuration, the detected abnormality can be displayed.

[0109] The image processing method of aspect 11 of the present invention performs an image acquisition process to acquire images of an object while moving along a predetermined route in each of a plurality of time periods; an image anomaly detection process to detect anomalies in the object based on the images taken in each time period; a point cloud generation process to generate point cloud data representing the object based on the images taken in the plurality of time periods; and a 3D anomaly detection process to detect anomalies in the object based on the point cloud data.

[0110] According to the above configuration, the same effects as those of the first aspect are achieved.

[0111] An image processing program according to aspect 12 of the present invention is an image processing program for causing a computer to function as the image processing device described in any one of aspects 1 to 9 above, and is an image processing program for causing a computer to function as each of the above parts.

[0112] According to the above configuration, the same effects as those of the first aspect are achieved.

[0113] The image processing device according to each aspect of the present invention may be realized by a computer. In this case, the image processing program of the image processing device that realizes the image processing device by the computer by operating the computer as each part (software element) of the image processing device, and the computer-readable recording medium on which it is recorded, also fall within the scope of the present invention.

[0114] The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. The technical scope of the present invention also includes embodiments obtained by appropriately combining the technical means disclosed in the different embodiments. Furthermore, new technical features can be formed by combining the technical means disclosed in the respective embodiments. [Explanation of symbols]

[0115] 10 Image display device 12 Storage section 13 Display section 20 Image processing device 21 Image acquisition unit 22 Point cloud generator 23 3D Deformation Detection Unit 24 Image Deformation Detection Unit 30 Imaging device

Claims

1. an image acquisition unit that acquires captured images of an object while moving along a predetermined route in each of a plurality of time periods; An image anomaly detection unit that detects anomalies in the target based on the captured images captured in each time period; a point cloud generation unit that generates point cloud data representing the object based on the captured images captured in the plurality of time periods; A three-dimensional deformation detection unit that detects deformation of the object based on the point cloud data.

13. An image processing device comprising:

2. The captured images captured in each time period include at least a captured image of the target captured from a position different from the captured images captured in time periods other than the time period.

2. The image processing device according to claim 1,

3. The captured images captured in each time period are images captured such that captured images captured before and after each time period overlap in part.

3. The image processing device according to claim 2.

4. When the captured images captured in each time period include an image that is not suitable for generating the point cloud data, the image acquisition unit supplements the captured image with another captured image that corresponds to the captured image.

2. The image processing device according to claim 1,

5. When the image anomaly detection unit detects an anomaly based on an image captured in a certain time period, the point cloud generation unit generates the point cloud data based on the captured images captured in a plurality of time periods after the certain time period.

2. The image processing device according to claim 1,

6. The point cloud generation unit generates point cloud data representing a deformation area including a deformation when the image deformation detection unit detects a deformation.

2. The image processing device according to claim 1,

7. An image generating unit that generates an ortho-image of the object based on the point cloud data and the captured image, The image generation unit generates each pixel of the orthoimage based on a pixel of the captured image captured in a specific time period among the multiple time periods.

2. The image processing device according to claim 1,

8. The image generating unit configures the orthoimage from a plurality of partial orthoimages, and sets the specific time period for each of the partial orthoimages.

8. The image processing device according to claim 7,

9. The three-dimensional deformation detection unit outputs information representing the position and degree of the detected deformation to a display device.

2. The image processing device according to claim 1,

10. An image processing device comprising:

1. An image display device comprising:

11. an image acquisition process for acquiring captured images of an object while moving along a predetermined route in each of a plurality of time periods; An image anomaly detection process for detecting an anomaly of the target based on the captured images captured in each time period; a point cloud generation process for generating point cloud data representing the object based on the captured images captured in the plurality of time periods; A three-dimensional deformation detection process is executed to detect a deformation of the object based on the point cloud data.

13. An image processing method comprising:

12. 10. An image processing program for causing a computer to function as the image processing device according to claim 1, the image processing program causing the computer to function as each of the above-mentioned units.

Images