Information processing device, information processing method, and program

Abstract

To provide an information processing device, an information processing method, and a program for determining whether to use a photographed image for predetermined image processing based on the quality of the image.SOLUTION: A method includes: performing an image quality determination for a photographed image based on a focusing degree, a frequency analysis result, or a photograph resolution; and determining whether the image is inappropriate for synthesis processing or processing for detecting a deformation that occurs in a photographing object, based on a result of the image quality determination (S205). In addition, the method includes identifying an image that is inappropriate for image processing of either synthesis processing or processing for detecting a deformation that occurs in a photographing object, as image that needs re-photographing (S208).SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a technique for determining the quality of a captured image. [Background technology]

[0002] In the inspection of buildings, methods for detecting cracks and other abnormalities from captured images require that the captured images are in focus and clear. Also, high-resolution images are required to detect small abnormalities within the inspection area, so when inspecting large structures, multiple high-resolution images are sometimes combined to create a single composite image.

[0003] Patent Document 1 discloses a technique for determining whether images to be used for synthesis have defects or other abnormalities, and specifying images that need to be re-photographed depending on whether or not there are any abnormalities. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6619761 Summary of the Invention [Problem to be solved by the invention]

[0005] As described above, captured images used in building inspections must be properly focused, have high resolution, and meet certain image quality requirements. If the captured images do not meet these standards, image processing required for inspection, such as anomaly detection and composition, cannot be performed correctly, and re-capture may be necessary. If the building is located far away, or if equipment for capturing images is required, re-capture can be a significant effort, especially if the image is captured over a different day. Therefore, there is a need for technology that can quickly determine whether captured images meet certain image quality requirements after being captured on-site, i.e., whether re-capture is necessary. Furthermore, it has been difficult for humans to appropriately determine the quality of such images visually.

[0006] The technology disclosed in Patent Document 1 identifies images to be re-photographed by detecting data abnormalities, such as missing image data, during communication of the captured image. However, if the image data is normal, it is not determined that re-photography is necessary even if the image quality is poor, such as if the recorded image is not properly focused or has low resolution. Therefore, the re-photography determination in the prior art does not take image quality into consideration, and there is room for improvement in this regard.

[0007] The present invention has been made in view of the above-mentioned problems, and one of the objects of the present invention is to determine whether a captured image should be used for predetermined image processing based on the image quality of the image. [Means for solving the problem]

[0008] The information processing device according to the present invention has the following configuration: Images containing structures Get Rutori With a means, a first determination means for performing a first determination as to whether or not the resolution of the image satisfies a predetermined condition; and when it is determined in the first determination that the resolution satisfies the predetermined condition, a first determination means for performing a first determination as to whether or not the resolution of the image satisfies a predetermined condition. The degree of focus meets the specified conditions The second judgment whether of and when it is determined in the second determination that a predetermined condition is satisfied, The aforementioned Image frequency components satisfies the specified conditions a third determination means for making a third determination as to whether the image satisfies a predetermined condition; a fourth determination means for determining that the image is suitable for predetermined image processing if it is determined in the third determination that the image satisfies a predetermined condition, and for determining that the image is not suitable for the predetermined image processing if it is determined in at least one of the first determination, the second determination, and the third determination that the image does not satisfy the predetermined condition; and a display control means for controlling the display of the image and the determination result of the fourth determination means. and, If it is determined in the first determination that the predetermined condition is not satisfied, the second determination and the third determination are not performed, and if it is determined in the second determination that the predetermined condition is not satisfied, the third determination is not performed. 1. An information processing device comprising: [Effects of the Invention]

[0009] According to the present invention, it is possible to determine, based on the image quality of a captured image, whether the image should be used for predetermined image processing. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 2 illustrates an example of a hardware configuration of an information processing device. [Figure 2] FIG. 2 is a diagram showing an example of a flowchart illustrating a processing flow of the information processing apparatus of the first embodiment. [Figure 3] FIG. 3 is a diagram showing an example of a flowchart of a photographing process according to the first embodiment. [Figure 4] FIG. 10 is a diagram showing an example of a captured image and a composite image. [Figure 5] FIG. 10 is a diagram showing an example of a photographed image list. [Figure 6] FIG. 10 is a diagram illustrating an example of a flowchart of a shooting condition determination process. [Figure 7] FIG. 10 is a diagram illustrating an example of a flowchart of image quality determination processing. [Figure 8] FIG. 10 is a diagram illustrating an example of a defocus map. [Figure 9] FIG. 10 is a diagram showing an example of an image to be re-captured. [Figure 10] FIG. 10 is a diagram showing an example of a flowchart of an image capturing process according to a first modified example of the first embodiment. [Figure 11] FIG. 10 is a diagram showing an example of a flowchart of image quality determination processing according to Modification 1 of Embodiment 1. [Figure 12] FIG. 10 is a diagram showing an example of a flowchart of re-photographing processing according to Modification 3 of Embodiment 1. [Figure 13] FIG. 13 is a diagram showing an example of a screen for confirming whether or not to perform re-imaging according to Modification 3 of Embodiment 1. [Figure 14] FIG. 10 is a diagram showing an example of a flowchart of image quality determination processing according to Modification 4 of Embodiment 1. [Figure 15] FIG. 10 is a diagram showing an example of a defocus map on which image quality determination results according to Modification 4 of Embodiment 1 are superimposed. [Figure 16] FIG. 13 is a diagram showing an example of a screen for selecting an image quality determination mode according to the fourth modification of the first embodiment. [Figure 17] 13 is a diagram showing an example of a defocus map on which the image quality determination results of Modification 4 of Embodiment 1 are superimposed (another example of superimposed display). DETAILED DESCRIPTION OF THE INVENTION

[0011] <Embodiment 1> FIG. 1 is a diagram of an information processing system including an information processing device according to this embodiment. As shown in FIG. 1, the information processing system is composed of an information processing device 100, a photography assist device 150, and a photography device 180. The information processing system implements an information processing method that judges the quality of a photographed image and determines whether or not the image needs to be re-photographed. The information processing system is also a system for inspecting a structure based on photographed images of the structure. Examples of structures include bridges, tunnels, roads, buildings, dams, levees, electrical equipment, etc.

[0012] The information processing device 100 is a device for controlling the overall imaging processing of this embodiment. It has a CPU 101, a ROM 102, a RAM 103, a HDD 104, a display unit 105, an operation unit 106, and a communication unit 107. The CPU 101 is a central processing unit (Central Processing Unit) that performs calculations and logical decisions for various processes and controls each component connected to a system bus 110. The ROM (Read-Only Memory) 102 is a program memory that stores programs for control by the CPU 101, including various processing procedures described below. The RAM (Random Access Memory) 103 is used as a temporary storage area such as the main memory and work area of ​​the CPU 101. Note that the program memory may be realized by loading a program into the RAM 103 from an external storage device connected to the information processing device 100.

[0013] The HDD 104 is a hard disk for storing electronic data such as image data and programs according to this embodiment. An external storage device may also be used to perform a similar function. Here, the external storage device can be realized, for example, by media (recording media) and an external storage drive for realizing access to the media. Known examples of such media include flexible disks (FDs), CD-ROMs, DVDs, USB memories, MOs, and flash memories. The external storage device may also be a server device connected via a network.

[0014] The display unit 105 is, for example, a liquid crystal display (LCD) or an organic light-emitting diode (OLED), and is a device that outputs images to a display screen. The display unit 105 may be an external device connected to the information processing device 100 by wire or wirelessly. The operation unit 106 has a keyboard and a mouse and accepts various operations by a user. The communication unit 107 performs wired or wireless two-way communication with other information processing devices, communication devices, external storage devices, etc. using known communication technologies. The communication unit 107 may be configured, for example, by a chip or antenna for public wireless communication. The communication unit 107 may also be configured to perform communication by other wireless communication methods, such as wireless LAN or Bluetooth (registered trademark).

[0015] In this embodiment, the photographing assist device 150 is a camera platform device that can change the photographing position and direction based on control from the information processing device 100, and is equipped with a photographing device 180, which will be described later. The photographing assist device 150 is also composed of a communication unit 151, a photographing position and direction control unit 152, and a photographing instruction unit 153. The communication unit 151 communicates with the information processing device 100 wirelessly or via a cable, and controls the photographing direction and position and gives photographing instructions according to instructions from the information processing device. The communication unit 151 may be composed of, for example, a chip or antenna for public wireless communication. Note that the communication unit 151 may also be configured to communicate via other wireless communication methods such as wireless LAN or Bluetooth (registered trademark).

[0016] The photographing position / direction control unit 152 changes the photographing position / direction of the camera platform device so that the photographing area of ​​the subject to be inspected can be photographed. The photographing instruction unit 153 controls the photographing device 180 set to the photographing position / direction changed by the photographing position / direction control unit 152 so that the photographing device 180 performs photographing.

[0017] The image capturing device 180 is a device that captures images based on image capturing instruction information received from the information processing device 100 via the image capturing auxiliary device 150. The image capturing device 180 has a full-field phase-difference imaging sensor, and also records focus level information (defocus value) of the captured image. The focus level information will be described in detail later using FIG. 8. The defocus value is data that represents the spatial (two-dimensional) distribution of defocus amounts in the capturing range. In the following explanation, the data that represents the spatial distribution of defocus amounts will also be referred to as a defocus map. The defocus amount is the amount of deviation from the distance at which the optical system of the image capturing device 180 is in focus. The defocus value (defocus map) represents the defocus amount as a spatial distribution for each pixel of the image.

[0018] Each pixel of the full-field phase-difference imaging sensor of the imaging device 180 has two photoelectric conversion units, which are referred to as divided pixel A and divided pixel B. In the full-field phase-difference imaging sensor, divided pixels A and B are arranged regularly in a two-dimensional manner and output images A and B as parallax images, respectively. Furthermore, image A+B, which is an image obtained by adding images A and B, is recorded as a recorded still image. The defocus amount is calculated based on the phase difference between these parallax images. Note that, although the description will be given taking as an example a configuration in which the defocus amount is derived pixel by pixel, the defocus amount may also be derived for each predetermined region, such as a block unit having multiple pixels (e.g., 5 pixels x 5 pixels).

[0019] Next, the photographing process according to this embodiment will be described. Fig. 2 is a flowchart showing the flow of processing by the information processing device 100 according to this embodiment. The flowchart shown in Fig. 2 starts when the information processing device 100 executes a photographing process control application. Hereinafter, each process (step) will be described with an S added to the beginning of the reference numeral.

[0020] In S201, the CPU 101 of the information processing device 100 performs a photographing process. The photographing process shown in S201 is a process for operating the photographing assist device 150 and the photographing device 180 and performing photographing under the control of the information processing device 100. The photographing process shown in S201 will be described later using the flowchart of FIG. 3. In S201, the photographing range and the photographing position are specified in the order shown in FIG. 4A, and the photographing assist device 150 is controlled and the photographing device 180 is caused to perform photographing. In S201, when the information processing device 100 receives a specification of an inspection range (photographing range) of a structure from the user, the information processing device 100 creates a table 501 shown in FIG. 5A, which is a photographed image list including information indicating the photographing range or the photographing position and direction of an image corresponding to each record. Furthermore, in S201, the information processing device 100 causes photographing to be performed in order based on the table 501. The information processing device 100 drives the photographing assist device 150 and controls the photographing range of the photographing device 180 based on the table 501, which is the photographed image list. When the information processing device 100 drives the photographing assist device 150 to set the photographing range, it transmits a photographing instruction to the photographing device 180 via the photographing assist device 150 to perform photographing. The photographing device 180 performs photographing in response to the photographing instruction. When photographing of one image is completed from the photographing assist device 150 or the photographing device 180, the information processing device 100 receives a photographing completion notification. Upon receiving the photographing completion notification, the information processing device 100 writes information such as the photographed image file name to the record in the photographed image list corresponding to that photographing. Then, in order to perform the next photographing, it transmits information for changing the photographing direction and a photographing instruction to the photographing assist device 150 and the photographing device 180, and performs photographing corresponding to the next record in the list. Photographing is repeated until photographing of images corresponding to the photographed image list created in this way is completed. When photographing of all images corresponding to the photographed image list is completed, the photographing process ends. After the photographing process ends, the photographed image file names are updated to the table 501 shown in FIG. 5(b) in which the photographed image file names are recorded. Although the information processing device 100 has been described as being configured to transmit information for driving the photography assist device 150 and photography instructions for each image, the present invention is not limited to this.For example, the information processing device 100 may be configured to transmit information for capturing all images corresponding to records included in the table 501, which is a list of captured images, all at once to the photographing assist device 150 and / or the photographing device 180. The information processing device 100 may also be configured to transmit information for capturing multiple images corresponding to records included in the table 501, which is a list of captured images, to the photographing assist device 150 and / or the photographing device 180. The photographing assist device 150 and / or the photographing device 180 may also be configured to transmit information to be input into the table 501, which is a list of captured images, all at once to the information processing device 100 when all the images have been captured.

[0021] Fig. 4(a) shows a photographing area 400 of the inspection target surface of the structure to be inspected. The photographing area 400 shows that the corresponding part of the inspection target surface is photographed in the direction indicated by the arrow 411, with the photographing position 421 indicated by the diamond-shaped rectangle as the center. Fig. 4(b) shows a composite image 450 made up of the images photographed in order in Fig. 4(a). The composite image 450 is used to inspect the photographing area 400, which is the inspection target surface.

[0022] In S202, the CPU 101 acquires the table 501 and the captured image file created in S201. From S203 to S205, the CPU 101 repeats the process for each acquired captured image file in the order of the captured image IDs listed in the table 501.

[0023] In S203, the CPU 101 performs a photographing condition determination process to determine the photographing conditions under which the photographed image was captured. The photographing condition determination process will be described later using the flowchart of FIG. 6. If the photographing conditions of the determined photographed image do not match the predetermined photographing conditions in the photographing condition determination process, the photographing condition is recorded as "out of photographing conditions," which is information indicating that the predetermined photographing conditions are not met, in the determination information field for the corresponding photographed image ID in table 501. The result of this photographing condition determination process is used in the re-photograph determination process, which will be described later. That is, according to this embodiment, it is possible to determine whether re-photographing is necessary based on the photographing conditions of the photographed image. Furthermore, by performing the determination process of S203 using the photographing conditions, it is possible to determine the photographed image that needs to be re-photographed without performing the image quality determination process of S205, which will be described later. Furthermore, by performing the photographing condition determination process of S203, which has a lower load than the image quality determination process of S205, which will be described later, at an earlier stage, the processing time can be shortened.

[0024] An example of the photographing condition determined in S203 is the aperture value (F-number). Regarding the aperture value (F-number), in order to use an image for inspection that has a deep depth of field and is less affected by blur due to diffraction, a range of conditions is set for the aperture value when the image to be used for inspection is captured. That is, in this embodiment, it is determined that a photographed area photographed with an aperture value outside a predetermined range or value should not be used for predetermined image processing such as abnormality detection and composition processing for inspection, and that re-photography is required.

[0025] An example of the photographing condition determined in S203 is the ISO value (sensitivity), which indicates the ability to capture light. Regarding the ISO value, if the ISO value is high and there is a possibility that the image will be affected by noise, the image will be unsuitable for inspection purposes. Therefore, a condition value is set for the ISO value at which the image to be used for inspection was captured. That is, in this embodiment, for a photographed range photographed at an ISO value outside a predetermined range or value, it is determined that the photographed range should not be used for predetermined image processing such as abnormality detection and composition processing for inspection, and that re-photography is required.

[0026] An example of the shooting condition determined in S203 is the subject distance. Regarding the distance to the subject at the time of shooting, if the distance is too far, the image resolution may be low, and small abnormalities may not be detected. Therefore, a condition is set for the distance to the subject when the image to be used for inspection is captured. That is, in this embodiment, for a shooting range captured at a subject distance outside a predetermined range or value, it is determined that the image should not be used for predetermined image processing such as abnormality detection and composition processing for inspection, and that re-shooting is required. Note that the shooting conditions determined in S203 may have conditions set for each of multiple attributes.

[0027] In S204, the CPU 101 uses the determination result of S203 to determine whether to proceed to the process of S205 for the image to be processed. If it is determined in S203 that the captured image matches the predetermined shooting conditions (Yes), the process proceeds to S205, and if it does not match (No), the process proceeds to S203 and processes the next captured image ID.

[0028] In S205, the CPU 101 performs an image quality assessment process to assess the image quality of the captured image. The image quality assessment process will be described later using the flowchart in FIG. 7. The information processing device 100 records the assessment results of the image quality assessment process in S205 in the assessment information field of table 501 shown in FIG. 5(c). In the image quality assessment process in S205, for example, focus information indicating the degree of focus for each predetermined region of the captured image is used to assess image quality. The reason for using focus information is that, for example, a gust of wind may occur during the image capture process, causing the image capture assist device 150 and the image capture device 180 to move, resulting in a shift in the focus position. Furthermore, if an obstruction such as another worker or a bird gets between the image capture device and the inspection target surface, the image may be captured at a focal length different from the actual focal length of the target, resulting in degradation of image quality. Images that are not properly focused on the inspection surface may not be used for inspection because image processing required for inspection, such as abnormality detection and composition, may not be performed correctly. Therefore, it is determined that recapture is necessary. Although the image quality determination process in S205 uses focus information to determine the image quality, it may be performed based on the resolution indicated by the number of pixels per unit length of the inspection surface.

[0029] Next, in S206, the CPU 101 determines whether the shooting condition determination process in S203 or the image quality determination process in S205 has been completed for all captured images input in S202. If the processing has been completed for all images (Yes), the process proceeds to S207, and if not (No), the process proceeds to S203, where the captured image with the next captured image ID is processed.

[0030] In S207, the CPU 101 determines whether the image quality determination result is NG or outside the shooting conditions in the determination information of the table 501. If the determination information includes an image quality determination result that is NG or outside the shooting conditions (Yes), the process proceeds to S208, and if not (No), the process ends.

[0031] In S208, the CPU 101 identifies the image to be re-photographed. In the first embodiment, the information processing device 100 creates a composite image based on the photographing positions and photographing ranges of each photographed image, and presents the photographing positions in the composite image that require re-photography to the worker. Images for which the determination information in the table 501 is not OK are identified, and the photographing positions and photographing ranges that require re-photography in the composite image are presented based on the photographing position information. As shown in FIG. 9, an image 921 to be re-photographed and an image 922 to be re-photographed in the composite image 450 are displayed on the display unit 105. Note that in the example of FIG. 9, the photographing positions or photographing ranges that require re-photography are presented to the user by highlighting them on the composite image, but other configurations are also possible. For example, the photographing positions and directions that require re-photography may be output as text information. Furthermore, information that allows identification of photographed images that require re-photography may be output. For example, the photographed images that require re-photography may be stored in a specified folder. Furthermore, information for identifying the presence of images that should not be used for image processing such as abnormality detection and composition may be output as a display or sound. Further, information indicating that there is an image that needs to be re-photographed may be output as a display or sound. Furthermore, information indicating the determination result in S205 may be output in association with each of the images that were the subject of the determination. For example, information indicating whether the image can / cannot be used for image processing such as abnormality detection or composition, or whether re-photography is / is not necessary, may be output in association with each of the images that were the subject of the determination. Furthermore, for example, information indicating "OK" indicating that the image quality meets the conditions, or information indicating "NG" indicating that the image quality does not meet the conditions, may be output in association with each of the images that were the subject of the determination. In this way, information indicating whether an image should be used for predetermined image processing may be output in various ways.

[0032] Next, the details of the photographing process shown in S201 in FIG. 2 will be described with reference to the flowchart shown in FIG. 3. In S301 in FIG. 3, the information processing device 100 receives a user specification of an inspection range (photographing range) for a structure. The information processing device 100 determines the number of photographed images and their respective photographing positions based on the area of ​​the specified inspection range (photographing range). For example, the information processing device 100 receives a specification from the user that the photographing target area 400 in FIG. 4(a) is the inspection range (photographing range). The information processing device 100 calculates the number of photographed images and their respective photographing positions based on two diagonal points of the photographing target area 400. For example, if the inspection target of a structure is 7 m wide x 3 m high and the area that can be photographed with one photographed image is 1 m wide x 1 m high, it is determined that 21 photographed images, arranged in 7 columns and 3 rows, are required, as shown in FIG. 4(b).

[0033] When the user has completed specifying the inspection range (photographing range), in S302, the CPU 101 creates a photographed image list shown in table 501 of FIG. 5 based on the number of photographed images and the photographing positions calculated in S301. In FIG. 5, table 501 is a table representing the photographed image list. Table 501 is composed of photographed image IDs, photographing positions, photographed image file names, and judgment information. Note that the attributes included in table 501 are merely examples, and not all of them are required. Furthermore, other attributes may also be included. The photographed image ID is an ID for identifying the photographed image file. The photographing position represents the photographing position used by the camera platform device. For example, the upper left corner is stored in a coordinate format such as "row 1, column 1." The judgment information stores the judgment result in the image quality judgment process in S205. When table 501 is created in S302, the photographed image ID and photographing position information calculated from the number of photographed images are entered, as shown in FIG. 5(a), and the photographed image file name and judgment information fields are blank.

[0034] In S303, the CPU 101 controls the camera platform device and the camera 180, which are the photography support device 150, to take photographs in the order of the photographed image IDs in the table 501 created in S302, based on the photographed image position information corresponding to the photographed image IDs.

[0035] The information processing device 100 changes the shooting direction and shooting position of the photographing assist device 150 for each photographed image ID based on the coordinate information described in the photographing position information in the table 501. Then, the information processing device 100 controls the photographing device 180 to, for example, adjust the focus using an autofocus function that uses the center of the screen as the focus point. The information processing device 100 transmits a shooting instruction to the photographing device 180 so that photographing is performed upon completion of autofocus. In response to the control of the photographing assist device 150 or a shooting completion notification transmitted from the photographing device 180, the information processing device 100 writes information such as the photographed image file name in the photographed image file name column of the table 501 in the record of the photographed image list, as shown in FIG. 5(b). The information processing device 100 repeats this process until all images corresponding to the photographed image list have been photographed. When all images corresponding to the created photographed image list have been photographed, the photographing process ends.

[0036] Next, the details of the shooting condition determination process shown in S203 of Fig. 2 will be described using the flowchart shown in Fig. 6. In S601 of Fig. 6, the CPU 101 acquires shooting information included in the captured image to be processed. The shooting information is metadata recorded as the shooting parameters used when the image was captured, and specifically includes the shooting time, aperture value (F-number), ISO sensitivity, etc.

[0037] In S602, the CPU 101 determines whether the values ​​of the shooting information acquired in S601 match the predetermined shooting conditions. If it is determined that the values ​​do not meet the conditions (Yes), the process proceeds to S603, and if it is determined that the values ​​meet the conditions (No), the process proceeds to end processing. The shooting conditions are thresholds and ranges that are set in advance, and the conditions are whether the values ​​included in the shooting information are within the threshold range. For example, the shooting conditions may be the aperture value, ISO, or distance to the subject described in the processing explanation of S203.

[0038] In S603, if the determination result in S602 is outside the shooting conditions, the CPU 101 determines that the determination result is outside the shooting conditions and records it in the determination information item of the table 501. Note that if the acquired shooting information value satisfies a predetermined shooting condition, the configuration may be such that this is recorded in the determination information item of the table 501.

[0039] Next, the details of the image quality determination process shown in S205 of Fig. 2 will be explained using the flowchart shown in Fig. 7. In S701 of Fig. 7, the CPU 101 acquires focus level information contained in the image to be processed. The focus level information is information acquired together with an image captured by an imaging device having a full-field phase-difference imaging sensor, and is information that records the amount of defocus for each region of the image.

[0040] In S702, the CPU 101 calculates the proportion of the in-focus area using the focus level information acquired in S701. The process of calculating the proportion in S702 will be described with reference to FIGS. 8(a) and 8(b). FIGS. 8(a) and 8(b) show an example of a defocus map, which is focus level information. Outer frames 801 and 821 correspond to the image size of the input image. Numerical values ​​802 and 822 represent defocus amounts indicating the focus level of each area. Lines 803 and 823 indicate boundaries between areas with different defocus amounts. Here, the defocus value is a quantified absolute value representing the amount of focus shift for each pixel of the image relative to the object. Note that the sign of the defocus amount may be different when the focus is shifted forward and backward in the depth direction. That is, for example, the defocus amount may be positive when the focus is shifted forward and negative when the focus is shifted backward.

[0041] When capturing an image, if an image plane phase difference imaging sensor is arranged on the entire surface of the sensor of the image capturing device 180, it is possible to acquire information on the defocus amount at the position of the pixel that can be acquired. Note that known techniques can be used to acquire the defocus value information. For example, autofocus techniques that use the amount of focus shift before and after detected from the imaging sensor are already widely used. Note that a configuration in which the defocus value is acquired using parallax images captured by a stereo camera or the like may also be used.

[0042] In the defocus map of Fig. 8, areas with a defocus value of "0" are areas that are in focus (areas that are in focus) in the focus unit of the image capture device 150. Also, areas with a defocus amount value greater than "0" indicate that the larger the value, the greater the amount of deviation from the distance at which the optical system is in focus (the worse the degree of focus). In the example of Fig. 8(a), the defocus amount is expressed in two stages, "0" and "1," and in the example of Fig. 8(b), it is expressed in three stages, "0," "1," and "2," but the number of stages is not limited to these and other values ​​may be used.

[0043] 8(a) will be used to explain the calculation of the proportion of the in-focus area. The calculation is performed by using the number of pixels surrounded by the outer frame 801 as the denominator and the number of pixels with a defocus value of "0" as the numerator. That is, the proportion of the in-focus area in the image is calculated. In the example shown in FIG. 8(a), the proportion of pixels with a defocus amount of "0" is about 90%, and in FIG. 8(b), the proportion of pixels with a defocus amount of "0" is about 60%.

[0044] In S703, the CPU 101 determines whether the ratio calculated in S702 is equal to or greater than a predetermined threshold. If the ratio is equal to or greater than the predetermined threshold (Yes), the process proceeds to S704. If the ratio is less than the threshold (No), the process proceeds to S705. If the ratio threshold is 80%, the ratio is equal to or greater than the threshold (80%) in FIG. 8(a), and is less than the threshold (80%) in FIG. 8(b). Note that although the threshold has been described as 80%, it may be set arbitrarily based on input by the user. The threshold may also be determined based on the relationship between the ratio of the in-focus area and the detection accuracy for detecting anomalies. For example, various methods may be employed, such as setting the threshold in advance to a ratio that will result in an average anomaly detection accuracy of 95% or greater.

[0045] Although S702 is configured to calculate the percentage of areas where the defocus amount is "0," this is not limiting. For example, it may be configured to calculate the percentage of areas where the defocus amount is "0" or "1." That is, it may be configured to calculate areas where the degree of focus is equal to or greater than a threshold. Also, in S702, the percentage of areas where the defocus amount is "3" may be calculated, and in S703, if the percentage is less than the threshold, the process may proceed to S704, and if the percentage exceeds the threshold, the process may proceed to S705. Also, in S702, a first percentage of areas where the defocus amount is "0" and a second percentage of areas where the defocus amount is "3" may be calculated. In this case, if the first percentage exceeds the first threshold and the second percentage is less than a second threshold different from the first threshold, the process may proceed to S704, and otherwise the process may proceed to S705. In this way, it is possible to determine whether the degree of focus of an image satisfies various predetermined conditions based on the focus degree information, and to determine whether the image should be used for image processing such as abnormality detection, composition, etc. It is also possible to determine whether the degree of focus of an image satisfies various predetermined conditions based on the focus degree information, and to use the result of this focus state determination to determine whether re-imaging is necessary.

[0046] Next, in S704, CPU 101 determines that the captured image file to be processed can be used for image processing such as abnormality detection and compositing, or that re-capture is not necessary (OK), records the determination result in the determination information item of table 501, and terminates the image quality determination process.

[0047] In S705, CPU 101 determines that the captured image file to be processed should not be used for image processing such as abnormality detection or compositing, or that re-capture is required (NG), records the result in the judgment information item of table 501, and terminates the image quality judgment process.

[0048] Although the image quality determination process in S205 uses focus level information, i.e., the degree of focus, in this example, image quality may be determined using the results of other image processing. For example, the degree of blur may be calculated from the distribution of high-frequency and low-frequency components using frequency analysis processing, and the image quality may be determined based on a predetermined threshold value for the intensity of the blur. That is, blur information indicating the degree of blur for each predetermined region of the image may be generated using image frequency analysis processing, and the proportion of the image in which the degree of blur satisfies a predetermined condition may be determined based on the generated blur information. Then, based on the determination result of whether the degree of blur satisfies the predetermined condition, determination information indicating whether the image should be used for predetermined image processing may be output.

[0049] Furthermore, the information indicating the degree of blur may be combined with the information on the degree of focus. That is, a configuration may be adopted in which it is determined whether or not re-shooting is necessary within the shooting range of the captured image using the result of determining image quality using the information on the degree of focus and the result of determining image quality using the degree of blur based on frequency analysis processing.

[0050] Although an example has been shown in which the photographing process of S201 is controlled and performed by the information processing device 100, the present invention is not limited to this. That is, the information processing device 100 may omit the process of S201 and, when acquiring photographed images in S202, acquire information necessary for subsequent processing, such as a photographed image list.

[0051] As described above, in this embodiment, the information processing device 100 determines the image quality of multiple captured images of a structure before using them to inspect the structure, such as by combining the captured images or detecting abnormalities. This allows images with poor image quality to be identified and presented to the worker as candidates for re-capture. Furthermore, by identifying the locations (positions) of captured images that need to be re-captured in the composite image, the worker can capture images at the identified capture positions, thereby reducing the effort required for re-capture. This prevents low-quality images from being included in the composite image, enabling more reliable detection of abnormalities such as cracks. Furthermore, since re-capturing is performed only on problematic images rather than all images, the process can be shortened.

[0052] Note that a configuration may be adopted in which visualization information visualizing the defocus value, as shown in FIG. 8, is generated for the captured image file for which image quality was determined in S205. For example, the visualization information visualizing the defocus value may be generated by superimposing information linked to the magnitude of the defocus amount on the captured image. The generated superimposed image may also be output in association with the image quality determination result. The superimposed information may be selected to have a darker color or a more saturated color as the defocus amount increases. For example, no information may be superimposed on pixels with a defocus amount of "0," while green may be superimposed on pixels with a defocus amount of "1," yellow on pixels with a defocus amount of "2," and red on pixels with a defocus amount of "3." By checking the generated superimposed image and visualizing the degree of focus using the superimposed information, the reason for the image quality determination being NG can be understood, enabling feedback for re-capturing. For example, after checking the superimposed image, if there is an obstruction such as a person or bird between the image and the surface to be photographed, or if a gust of wind has caused the focal distance of the image capturing device 180 to shift or its orientation to change during photography, it is sufficient to re-photograph the image under the same conditions. However, if this is not the case, for example, if there is a step on the inspection surface that causes the image to be out of focus, the aperture value in the photography settings of the image capturing device 180 is changed so that the area of ​​the step falls within the depth of field, and then photography is performed again. In this way, by generating an image in which the degree of focus is visualized, it is possible to easily determine the reason for the out-of-focus state and appropriately perform re-photography.

[0053] Furthermore, in this embodiment, the imaging control device 150 is an example of a camera platform device. However, the imaging device 180 may be mounted on a drone (unmanned aerial vehicle) capable of autonomous flight, and imaging may be performed while changing the imaging position. When using a drone, the imaging position and direction specified by the operator are measured using an onboard GNSS (Global Navigation Satellite System) device, altimeter, electronic compass, etc., and imaging is performed at the specified position. Furthermore, when imaging is performed in a location where a GNSS device cannot be used, such as the floorboards of a bridge, the imaging position is measured by communicating with a base station installed on the ground and measuring the direction and distance relative to the drone. Note that technology related to measuring the imaging position is not the focus of the present invention, and therefore will not be described here.

[0054] In addition, in the system according to the present embodiment, the information processing device 100, the photography control device 150, and the photography device 180 are configured as separate, independent devices, but they may also be a single device having the functions of each device. For example, they may be an integrated drone device with photography functions. Furthermore, they may also be configured to perform distributed processing using even more devices.

[0055] <Modification 1 of Embodiment 1> In the first embodiment, a configuration has been described in which the identified image to be re-photographed is displayed on a composite image as shown in Fig. 9, and the worker can re-photograph the image based on the displayed information. In the first modified example of the first embodiment, instead of displaying the position of the image to be re-photographed on the screen, an example of processing will be described in which the captured image is moved to a different directory based on the result of image quality judgment, and the image to be re-photographed is presented to the user.

[0056] FIG. 10 is a flow showing the flow of the photographing process in variant example 1 of embodiment 1, which replaces the flow in FIG. 3, which is S201 of the main flow of embodiment 1. The processes with the same step numbers are the same as those in FIG. 3, so their explanations will be omitted.

[0057] 10, the information processing device 100 first performs the above-described processes of S301 to S303 as the photographing process. Next, in S1001, the CPU 101 creates a directory named OK and a directory named NG in the directory containing the photographed image file.

[0058] Next, we will explain the image quality determination process in Modification 1. Fig. 11 is a flowchart showing the flow of the image quality determination process in Modification 1 of Embodiment 1. The same step numbers are used for processes similar to those shown in the flowchart in Fig. 7 above, and details thereof will be omitted.

[0059] In FIG. 10, the processes of S701 to S705 are the same as those described above, but differ from the flowchart of FIG. 7 in that S1101 is added after the process of S704 and S1102 is added after the process of S705.

[0060] In step S1101, which is performed after the processing in step S704, the CPU 101 moves the captured image file determined to be OK in step S704 to the OK directory created in step S1001.

[0061] In step S1102, which is performed after the processing in step S705, the CPU 101 moves the captured image file determined as NG in step S705 to the NG directory created in step S1001.

[0062] The above is a flowchart of the image quality determination process (S205) in Modification 1 of Embodiment 1. According to Modification 1, it is possible to determine which files are to be re-captured based on whether or not there are captured image files in the NG directory. Furthermore, in the shooting condition determination process of S203, captured image files that are determined not to satisfy the shooting conditions are not moved to another directory, and therefore can be determined to be files to be re-captured. Note that a configuration may be adopted in which captured image files that are determined not to satisfy the shooting conditions are moved to the NG directory as files to be re-captured. Furthermore, a shooting condition NG directory may be created separately in S1001, and captured image files that are determined not to satisfy the shooting conditions are moved to that directory.

[0063] Furthermore, by classifying the images into directories, if all the captured image files have been moved to the OK directory, the image quality of all the captured images will be good, and inspection work using the composite image can be started immediately.

[0064] Furthermore, by adopting a configuration in which images are classified into directories according to the first modification of the first embodiment, it becomes possible to check images together for each image quality judgment result. The worker can immediately check images that are judged to require re-shooting, or check for poor image quality by visually checking the superimposed image file that visualizes the focus level information described in the first embodiment.

[0065] Furthermore, by moving and classifying the captured image files into directories, the captured image files that were judged to be NG are separated out and the file is made up of only files that were judged to be OK, eliminating the need for workers to classify images. Also, even if re-shooting is performed, all of the images necessary for compositing are in the OK directory, and by adding the re-shooting images that were judged to be OK, it is possible to immediately generate the composite image in the subsequent stage or perform inspection using the composite image.

[0066] As described above, according to the first modification, the information processing device 100 outputs information for storing a file representing the image in a predetermined directory based on the result of the image quality determination process. Then, the storage destination of the file representing the image makes it possible to indicate the need for re-shooting.

[0067] <Modification 2 of Embodiment 1> In the first embodiment, the list of images to be re-captured is managed on the RAM 103 or the HDD 104. In the second modification of the first embodiment, an example of processing will be described in which the list of captured images in the table 501 is output in a file format rather than being transmitted in memory.

[0068] In Modification 2, the processing for creating table 501, which is a list of captured images, is changed in S302 of Fig. 3, which is the shooting processing of S201 in the main flow of Fig. 2 in Embodiment 1. When table 501 is created, a list file is created in CSV format, in which items are separated by commas, and saved in a directory containing the captured images to be input. Note that the captured image (file name) and judgment information items in the list file to be created are created blank, and are added when the captured image file is created in S303, and are added when the results of the shooting condition judgment in S203 and the image quality judgment in S205 are obtained.

[0069] With this configuration, the list of captured image files to be re-captured can be treated as a list file, making it possible to list the judgment results for the captured image files. Furthermore, by making it a list file, it can be used in other devices, other systems, and other applications.

[0070] <Third Modification of First Embodiment> In the first embodiment, the first modification of the first embodiment, and the second modification of the first embodiment, the worker confirms (visually confirms) the determination result of whether or not the object is to be re-photographed, and then re-photographs are taken. In the third modification of the first embodiment, an example will be described in which the determination result and information on the photographing position where re-photographing is necessary are used to control the photographing auxiliary device 150 and perform re-photographing.

[0071] Fig. 12 is a flow chart showing the flow of the re-photographing process. The re-photographing process in Fig. 12 replaces the re-photographing specification process in S208 in Fig. 2. The re-photographing process may also be configured to be performed after S208 in Fig. 2.

[0072] In S1201, the CPU 101 extracts from the table 501 the captured image IDs whose determination information is not OK, that is, NG, which indicates that re-capture is required.

[0073] In S1202, the CPU 101 acquires information about the photographing position of the photographed image ID extracted in S1201.

[0074] In S1203, similar to S303 in Fig. 3, the CPU 101 controls the designated photography assist device 150 using the information on the photography position acquired in S1202 to perform photography. This completes the flowchart of the re-photographing process using Fig. 12.

[0075] In the third modification of the first embodiment, the photographing position is identified from the photographed image ID of the image to be re-photographed, and the photographing support device 150, which is a camera platform device, is controlled using information on the identified photographing position to perform the re-photographing. This makes it possible to perform the re-photographing without the operator having to perform any operation for the re-photographing.

[0076] In addition, the captured image file captured in the re-capture processing of Figure 12 may be input as the input target of S202 in Figure 2, and the flowchart of Figure 2 may be executed in sequence to determine whether the re-captured captured image file needs to be re-captured.

[0077] 13 in S1203, and executes the re-shooting process based on the selection instruction of the operator. This not only automatically executes the re-shooting, but also checks the captured image file that caused the re-shooting and checks the cause of the image that was judged NG in image quality, thereby making it possible to avoid another NG judgment when re-shooting.

[0078] <Fourth Modification of First Embodiment> In the first embodiment and the first, second, and third modifications of the first embodiment, focus level information is used in the image quality determination process. As described above, image quality may be determined using the degree of blur based on frequency analysis processing instead of focus level information, or the result of determining image quality using focus level information may be combined with the result of determining image quality using the degree of blur based on frequency analysis processing. In this modification, an example will be described in which, in addition to determination using focus level information, determination is performed using imaging resolution information indicating the imaging resolution and frequency analysis information indicating the frequency analysis result.

[0079] Shooting resolution is the size of the shooting surface per pixel in the captured image, and is expressed in mm / pixel. Shooting resolution can be calculated from the size of the imaging sensor, the image size of the shooting surface, and the distance information from the shooting surface. As the shooting resolution value increases, the resolution becomes coarser, making it more difficult to see defects such as cracks.

[0080] Frequency analysis information is the calculation of the average value of the frequency components obtained by performing frequency analysis processing on the image. If the calculated average value is small, there are few high frequency components, that is, few sharp edges, and therefore there is a lot of blur and shake (a strong degree of blur and shake). In addition to determining blur when judging focus, frequency analysis can also be used to determine blur, making it possible to exclude images in which the effects of blur make it difficult to see cracks or other abnormalities.

[0081] 14 is a flow chart showing the flow of the image quality determination process. The image quality determination process in FIG. 14 replaces the image quality determination process in FIG.

[0082] In S1401, the CPU 101 acquires the shooting resolution information contained in the captured image. More specifically, the CPU 101 calculates the shooting resolution from the image size of the captured image, the size of the imaging sensor, and the distance from the shooting surface. The distance from the shooting surface is obtained by acquiring the distance to the subject when focusing at the position of the subject's range-finding point.

[0083] In S1402, the CPU 101 determines whether the image capture resolution acquired in S1401 is equal to or less than a predetermined threshold. If it is equal to or less than the threshold (Yes), the process proceeds to S1403, and if it is greater than the threshold (No), the process proceeds to S705. This makes it possible to determine whether the image capture resolution is what is required for the quality of the inspection image. If the image capture resolution is greater than the threshold, that is, if the subject is captured roughly, the image is determined to be NG without needing to perform the subsequent determination process. The setting of the threshold for determining the image capture resolution will be described later with reference to a setting screen 1601 in FIG. 16.

[0084] The processes of S701, S702, and S703 are the same as those described with reference to FIG. 7, and therefore will not be described here.

[0085] In S1403, the CPU 101 determines whether to continue the processing after the shooting resolution determination. If it continues (Yes), the process proceeds to S701, and if it ends, the process proceeds to S704. The setting of whether to continue the processing after the shooting resolution determination will be explained later with reference to a setting screen 1601 in FIG. 16.

[0086] In S1404, the CPU 101 determines whether or not to continue the processing after the focus level determination processing. If the processing is to be continued (Yes), the process proceeds to S1405, and if the processing is to be ended, the process proceeds to S1704. As in S1403, the setting of whether or not to continue will be explained later with reference to the setting screen 1601 in FIG. 16.

[0087] In S1405, the CPU 101 performs frequency analysis on the captured image to obtain frequency component values. As an example of the frequency analysis, a wavelet transform is used to calculate horizontal and vertical frequency components. The average value of the obtained frequency components is calculated, and this calculated value is used as the frequency component value.

[0088] In S1406, the CPU 101 determines whether the frequency component value calculated in S1403 is less than a predetermined threshold. If it is less than the threshold (Yes), the process proceeds to S704. If it is equal to or greater than the threshold (No), the process proceeds to S705. As a result, if the frequency component value is greater than the threshold, it is determined that there are many high-frequency components and many areas containing edges. If the frequency component value is less than the threshold, it is determined that there are many low-frequency components and that the effects of blur and shake are significant. By performing blur determination processing after determining the focus level, it is possible to determine whether an image is suitable for inspection even in a shooting situation where it is determined to be in focus but blur is present. An example of a situation where blur occurs even when an image is in focus is when the image is taken from an aircraft such as a drone, and the drone is expected to shake due to wind or other factors during shooting. The blur determination threshold will be explained later with reference to the setting screen 1601 in Figure 16.

[0089] In S1407, the CPU 101 generates an image on which the determination result is superimposed using information on the determination result in S704 or S705. The generated image will be described with reference to FIG.

[0090] Resultant image 1500 is composed of defocus region 1501, defocus region 1502, defocus region 1503, and frame region 1511. Frame region 1511 is composed of shooting resolution determination result region 1512, focus level determination result region 1513, and blur determination result region 1514. Defocus regions 1501, 1502, and 1503 are regions obtained by dividing the defocus map shown in FIG. 8(b) by defocus amount, with the region where the defocus amount is 0 being represented as defocus region 1501. Similarly, the region where the defocus amount is 1 is represented as defocus region 1502, and the region where the defocus amount is 2 is represented as defocus region 1503, and regions with the same defocus amount are treated as the same defocus region. A colored layer is generated for each defocus region, and the colored layer is superimposed on the captured image. The layer may be colored in a warning color, such as blue or green to indicate safety when the defocus amount is smaller than a predetermined value, or yellow or red to indicate caution when the defocus amount is larger than the predetermined value. This makes it easier to see areas where the defocus amount is large and re-imaging is required.

[0091] The frame area 1511 displays a shooting resolution determination area (shooting resolution determination icon) 1512, a focus level determination area (focus level determination icon) 1513, and a blur determination area (blur determination icon) 1514 as marks representing each determination result. The result image 1500 shows an example in which the shooting resolution determination and focus level determination are OK, but the blur determination is NG, with different representations used depending on the result. As an example of icon representation, the shooting resolution determination area 1512 and focus level determination area 1513 that are OK are represented by black text against a white background, while the blur determination area 1514 that is NG is represented by white text against a black background. This not only allows the determination result to be visually confirmed, but also makes it easy to determine at which stage an error occurred. The presence or absence of an icon may also indicate which determination method was not executed. For example, a case will be described in which the CPU 101 determines in S1402 of the image quality determination flow of FIG. 14 that the shooting resolution acquired in S1401 is equal to or lower than a predetermined threshold. In this case, only the imaging resolution determination area 1512 may be displayed in the frame area 1511 of the resultant image 1500, and the focus level determination area 1513 and the blur determination area 1514 may not be displayed.

[0092] The color of each icon in the frame area 1511 may also be changed based on the judgment result. For example, red is assigned to the shooting resolution judgment, blue to the focus judgment, and yellow to the blur judgment, and if an image is judged NG in the focus judgment, an image is created in which the frame area is filled in blue. This makes it possible to distinguish judgment results that are difficult to distinguish using the icons in the judgment area, even for small image sizes such as thumbnail images, by coloring the frame area. This concludes the explanation of the flowchart of image quality judgment processing shown in Figure 14.

[0093] FIG. 16 shows a setting screen 1601, which is an example of a UI screen for setting parameters in this modification. The setting screen 1601 includes a determination pattern selection section 1611 and a superimposed image saving condition selection section 1630. The determination pattern selection section 1611 is divided into three sections: a shooting resolution determination section 1612, a focus level determination section 1613, and a blur determination section 1614. Each determination section has a determination name label 1615, 1616, 1617, and a determination symbol 1618, 1619, 1620, respectively. Input fields 1621 to 1626 for values ​​such as determination thresholds used in the image quality determination flow of FIG. 14 are arranged within 1612 to 1614, respectively, corresponding to the determination processes that use them. More specifically, the shooting resolution determination section 1612 includes a shooting resolution determination name label 1615, a shooting resolution determination symbol 1618, and a shooting resolution determination threshold input field 1621. The focus level determination section 1613 is provided with a focus level determination name label 1616, a focus level determination symbol 1619, a focus level determination threshold input area 1622, a focus level determination ratio input area 1623, and a focus level determination area input area 1624. The blur determination section 1614 is provided with a blur determination name label 1617, a blur determination symbol 1620, a blur determination threshold input area 1625, and a blur determination area input area 1626. This arrangement allows the user to intuitively understand which determination process each numerical value relates to.

[0094] Furthermore, by matching the shapes of icons 1512, 1513, and 1514 in Fig. 15 with judgment symbols 1618, 1619, and 1620, respectively, the user can intuitively grasp the relationship between the judgment settings made on this screen and the results on the result superimposed image. Of course, these icons and judgment symbols may have different designs. The setting screen 1601 is a screen displayed on a PC, smartphone, tablet terminal, etc., and the user can turn check boxes and radio buttons (described below) on and off and enter numerical values ​​by mouse operation, touch operation, keyboard operation, etc. (using the operation unit 106).

[0095] Of the judgment name labels, focus level judgment name label 1616 consists of a focus level judgment checkbox 1616a and a focus level judgment name section 1616b. Blur judgment name label 1617 consists of a blur judgment checkbox 1617a and a blur judgment name section 1617b. By turning these checkboxes on or off, you can specify whether to further perform the corresponding judgment process when the result of the shooting resolution judgment, which is always performed, is OK. For example, in Figure 16, focus level judgment checkbox 1616a is ON and blur judgment checkbox 1617a is OFF, indicating that when the result of the shooting resolution judgment is OK, focus level judgment will continue and blur judgment will not be performed.

[0096] 16, the focus level determination checkbox 1616a is ON, so input fields 1622 to 1624 are available for input. In addition, the blur determination checkbox 1617a is OFF, so input fields 1625 and 1626 are grayed out, indicating that input is unavailable. This allows the user to focus only on inputting the numerical values ​​related to the determination to be made.

[0097] The CPU 101 determines whether to continue processing in S1403 and S1404 based on the status of the focus level determination checkbox 1616a and the blur determination checkbox 1617a. This allows the subsequent processing to be omitted even if the previous processing is determined to be OK if the subsequent processing is not required. For example, when using a drone, which is prone to blur, the blur determination checkbox 1617a can be turned ON to perform blur determination. When using a tripod, which is less prone to blur, the blur determination checkbox 1617a can be turned OFF to prevent blur determination. Furthermore, when only determining the image resolution, turning OFF the focus level determination checkbox 1616a and the blur determination checkbox 1617a allows the user to know the image resolution and confirm whether an image suitable for inspection has been captured. When performing determination processing on a large number of images, the processing can be terminated depending on the application and purpose, which is expected to reduce the processing time as described above.

[0098] The value input into the imaging resolution determination threshold input area (first setting means) 1621 represents the threshold value in S1402. The value input into the focus level determination threshold input area 1622 represents a predetermined value in S702. The value input into the focus level determination ratio input area (second setting means) 1623 represents the ratio threshold in S703. The value input into the focus level determination region input area (second setting means) 1624 is an item for setting the region for calculating the ratio in S702, and represents the percentage of the area of ​​the central region to be determined when the area of ​​the entire image to be determined is 100%. By reducing the central region to be determined (narrowing the area), the amount of calculation processing can be reduced, and speed can be improved. Furthermore, by setting the region to be determined to less than 100% (for example, 50%) using the value input into the focus level determination region input area (second setting means) 1624, only the central portion of the image to be applied to the composite image will be determined when creating a stitched composite image. By only targeting the center, the outer edges of the image that will become overlapping areas are excluded from the target of judgment. This makes it possible to determine that an image can be used for stitching even if the outer edges are out of focus.

[0099] The value input into the blur determination threshold input area (third setting means) 1625 represents the threshold value in S1406. The value input into the blur determination area input area (third setting means) 1626 is an item for setting the area for calculating frequency component values ​​in S1405, and represents the percentage of the area of ​​the central area to be calculated when the area of ​​the entire image to be calculated is 100%. In addition to having the same effect as the focus determination area, in blur determination, when the structure to be inspected is a bridge pier, the effect of suppressing the mixing of high-frequency components that occur when plants or the like are reflected on the back side of the structure can be obtained.

[0100] In this modification, the image quality determination process of S205 is replaced with the flowchart shown in FIG. 14, with the imaging resolution determination performed before the focus determination, and the blur determination process performed after the focus determination. The imaging resolution determination makes it possible to determine whether the image is suitable for capturing an inspection image, shortening the processing time and enabling a large number of images to be evaluated in a short time. Furthermore, by performing the blur determination, it is possible to determine the effect of blur when shooting using a drone, where blur occurs even when the image is determined to be in focus. Furthermore, by selecting whether or not to perform the determination process when each determination process is NG, the processing time can be shortened depending on the shooting environment.

[0101] In this modified example, the processes are performed in the order of image capture resolution determination, focus level determination, and blur determination as shown in FIG. 14, but any order may be used. However, performing the processes in the order shown in FIG. 14 provides the following advantages. By performing image capture resolution determination first, it is possible to determine whether to continue or terminate the process based on whether the image capture resolution of the image is suitable for inspection. Furthermore, by performing blur determination processing last, it is possible to determine whether an image is blurred even if it is in focus. Furthermore, by including a time-consuming frequency analysis process at the final stage, the number of times the frequency analysis process is executed can be reduced based on the determination results of the previous stages, thereby shortening the overall processing time.

[0102] In the screen of Figure 16, the order of the shooting resolution determination unit 1612, focus level determination unit 1613, and blur determination unit 1614 allows the user to intuitively understand that these three determination processes are performed in the order listed on the setting screen 1601.

[0103] Furthermore, the imaging resolution determination unit 1612 is not provided with checkboxes such as the focus level determination checkbox 1616a and the blur determination checkbox 1617a. This is because if the imaging resolution is not what is required for the quality of the inspection image, that is, if the subject is photographed roughly, there is no need to perform the subsequent determination process.

[0104] Furthermore, the focus level determination checkbox (first selection means) 1616a and the blur determination checkbox (second selection means) 1617a may be linked so that when both are OFF and the blur determination checkbox 1617a is turned ON, the focus level determination checkbox 1616a is also turned ON, and when both are ON and the focus level determination checkbox 1616a is turned OFF, the blur determination checkbox 1617a is also turned OFF.

[0105] This makes it possible to ensure that blur determination is performed after determining that the image is in focus. The reason for adopting such a setting is that, as mentioned above, by performing blur determination processing after determining the degree of focus, it is possible to determine whether the image is suitable for inspection even in shooting situations where blur occurs, even if the image is determined to be in focus.

[0106] Regarding the generation of the judgment result superimposed image, in this example it is created regardless of whether the judgment result is OK or NG, but it is also possible to generate a superimposed image only for objects judged as NG, or to shorten the processing time without generating a superimposed image by creating a list of the judgment results. The superimposed image saving condition selection section 1630 in FIG. 16 is an example of such selection using radio buttons. In FIG. 16, only "NG" is selected, but "Do not save" or "Save all" may also be selected depending on the user's operation.

[0107] When “Do not save” is selected in the superimposed image saving condition selection section 1630, the judgment process shown in FIG. 14 is performed, and images judged as OK are stored in a predetermined folder, and images judged as NG are stored in a separate folder, but an image with the image quality judgment result superimposed as shown in FIG. 15 is not created. When only “NG” is selected, the judgment process shown in FIG. 14 is performed, and images judged as OK are stored in a predetermined folder, and images judged as NG are stored in a separate folder. In addition, an image with the image quality judgment result superimposed as shown in FIG. 15 is also stored in the folder where the images judged as NG are stored. When “Save all” is selected, the judgment process shown in FIG. 14 is performed, and images judged as OK are stored in a predetermined folder, and images judged as NG are stored in a separate folder. In addition, an image with the image quality judgment result superimposed as shown in FIG. 15 is also stored in the folder where the images judged as OK are stored, and an image with the image quality judgment result superimposed as shown in FIG. 15 is also stored in the folder where the images judged as NG are stored. By adopting such settings, it is possible to generate only the necessary images that meet the user's needs while shortening the processing time.

[0108] In addition, in S1407, regarding the generation of the judgment result superimposed image, an icon indicating the judgment result is superimposed on the result image, so that it is possible to know in which process the judgment was NG. Furthermore, since no icon is displayed for processes that did not perform judgment processing, it is possible to know to what extent the processing has been performed.

[0109] Also, an example of another display form of the resultant superimposed image will be described with reference to Fig. 17. However, explanations of the same numbers as in Fig. 15 will be omitted, and only differences will be described.

[0110] 17(a) shows an example of an image that has been determined to be NG in the blur determination process as a result of performing imaging resolution determination, focus determination, and blur determination. A left frame area 1711 displays an imaging resolution information area 1721, imaging resolution determination area 1731, focus determination area 1732, and blur determination area 1733, while a right frame area 1712 displays an NG determination area 1741. The imaging resolution information area 1721 displays the value of the imaging resolution information acquired in S1401 (0.5 in FIG. 17(a)).

[0111] 17(a), three determinations have been made as described above, and so the display shows a shooting resolution determination area 1731, a focus level determination area 1732, and a blur determination area 1733. The shooting resolution determination area 1731, focus level determination area 1732, and blur determination area 1733 are each displayed in a different color, and in this modified example, the shooting resolution determination area 1731 is assigned red, the focus level determination area 1732 is assigned blue, and the blur determination area 1733 is assigned yellow.

[0112] The same color as the judgment area corresponding to the judgment process that resulted in NG is displayed in the NG judgment area 1741 in the right frame area 1712. In FIG. 17(a), the blur judgment process is NG, so the NG judgment area 1741 is displayed in the same color as the blur judgment area 1733.

[0113] 17(b) shows an example of a result image 1750 obtained when the result of the photographing resolution determination and focus level determination was NG in the focus level determination process, and no blur determination was performed. In FIG. 17(b), the left frame area 1711 displays the photographing resolution determination area 1731 and focus level determination area 1732, but does not display the blur determination area 1733. This is because, as described above, the photographing resolution determination and focus level determination were NG in the focus level determination process, and no blur determination was performed. The NG determination area 1741 in the right frame area 1712 displays the same color as the focus level determination area 1732, which corresponds to the NG focus level determination.

[0114] 17(c) is an example of a result image 1770 in which no NG judgment is made as a result of performing the shooting resolution judgment, focus judgment, and blur judgment. In FIG. 17(c), a shooting resolution judgment area 1731, a focus judgment area 1732, and a blur judgment area 1732 are displayed in the left frame area 1711, and an NG judgment area 1741 is not displayed in the right frame area 1712.

[0115] If the shooting resolution judgment is NG, only the shooting resolution judgment area 1731 is displayed in the left frame area 1711, and the same color as the shooting resolution judgment area 1731 is displayed in the NG judgment area 1741 in the right frame area 1712. Furthermore, if only the focus level judgment check box 1616a is turned ON in the judgment pattern selection unit 1611 and neither the shooting resolution judgment nor the focus level judgment is NG, the shooting resolution judgment area 1731 and the focus level judgment area 1732 are displayed in the left frame area 1711, and nothing is displayed in the NG judgment area 1741 in the right frame area 1712.

[0116] This UI allows the user to check the shooting resolution information in the result image. Furthermore, the user can check which of the three determination processes (shooting resolution determination, focus level determination, and blur determination) was actually performed, and intuitively check which process was found to be NG. In other words, it is possible to generate visualization information that visualizes which of the shooting resolution determination, focus level determination, and blur determination processes was performed, and which of the performed determinations was determined not to satisfy a predetermined condition.

[0117] Furthermore, by making the colors of the shooting resolution determination area 1731, focus level determination area 1732, and blur determination area 1733 correspond to the colors of the determination symbols of the shooting resolution determination symbol 1618, focus level determination symbol 1619, and blur determination symbol 1620 in Figure 16, it becomes possible to intuitively grasp the relationship between the determination set by the user and the result even on the resulting superimposed image.

[0118] Furthermore, in this modified example, an example of an information processing device that includes a determination unit having the functions of determining focus level, blur, and image capture resolution and that can select a determination mode has been described. However, the determination unit may be configured to have only one or two of these functions. For example, the determination unit may be configured to perform only the image capture resolution determination function, or may be configured to perform only the focus level determination and image capture resolution determination functions. The configuration of the acquisition unit may be matched to the configuration of the determination unit. Even with this configuration, the information processing device can determine whether a captured image should be used for predetermined image processing based on the image quality of the image.

[0119] <Other embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0120] 100 Information processing device 150 Shooting control device 180 Imaging Device 400 Shooting Area 411 Arrow 450 composite images 801 Outer Frame 802 Number 803 line 821 Outer Frame 822 Number 823 line 921~922 Re-photographed images

Claims

1. An acquisition means for acquiring an image including a structure; a first determination means for performing a first determination as to whether or not the resolution of the image satisfies a predetermined condition; a second determination means for performing a second determination as to whether or not a degree of focus for each predetermined region of the image satisfies a predetermined condition when it is determined in the first determination that the predetermined condition is satisfied; a third determination means for performing a third determination as to whether or not a frequency component of the image satisfies a predetermined condition when it is determined in the second determination that the predetermined condition is satisfied; a fourth determination means for determining that the image is suitable for predetermined image processing when it is determined in the third determination that the image satisfies a predetermined condition, and for determining that the image is not suitable for the predetermined image processing when it is determined in at least one of the first determination, the second determination, and the third determination that the image does not satisfy a predetermined condition; a display control means for controlling the display of the image and the determination result of the fourth determination means; and If it is determined in the first determination that the predetermined condition is not satisfied, the second determination and the third determination are not performed; If it is determined in the second determination that the predetermined condition is not satisfied, the third determination is not performed.

1. An information processing device comprising:

2. The acquisition means acquires shooting resolution information indicating a resolution of the image, 2. The information processing apparatus according to claim 1, wherein the first determination means determines whether the resolution is equal to or greater than a first threshold value based on the image capturing resolution information.

3. The acquisition means acquires focus degree information indicating a degree of focus for each predetermined area of ​​the image, 3. The information processing apparatus according to claim 1, wherein the second determination means determines, based on the focus level information, the proportion of an area in the entire image where the degree of focus exceeds a second threshold value.

4. 4. The information processing apparatus according to claim 3, wherein the focus level information is information indicating a defocus amount for each pixel of the image.

5. 5. The information processing apparatus according to claim 3, further comprising: a generating unit that generates visualization information for visualizing the degree of focus of the image based on the focus degree information.

6. The acquisition means acquires frequency analysis information that is a result of analyzing the frequency components of the image, 6. The information processing apparatus according to claim 1, wherein the third determination means determines whether the frequency component exceeds a third threshold value based on the frequency analysis information.

7. An information processing device as described in any one of claims 1 to 6, characterized in that when it is determined that a specified condition is not met in at least one of the first judgment, the second judgment, and the third judgment, the display control means outputs information indicating which judgment was determined to not meet the specified condition.

8. The information processing device described in Claim 2, characterized in that the display control means controls the display of information for setting the first threshold value based on user operation.

9. An information processing device described in any one of claims 3 to 5, characterized in that the display control means controls the display of information for setting the second threshold value based on user operation.

10. The information processing device described in Claim 6, characterized in that the display control means controls the display of information for setting the third threshold value based on user operation.

11. 11. The information processing device according to claim 1, wherein the display control means controls to display information indicating that re-shooting is required when it is determined that a predetermined condition is not met in at least one of the first judgment, the second judgment, and the third judgment.

12. 12. The information processing apparatus according to claim 11, wherein the display control means outputs information indicating a photographing position or a photographing range of the image as information indicating that re-photographing is required.

13. An information processing device as described in any one of claims 1 to 12, characterized in that it further has a storage means for storing the image in a first directory if it is determined in the third judgment that a predetermined condition is met, and for storing the image in a second directory different from the first directory if it is determined in at least one of the first judgment, the second judgment, and the third judgment that a predetermined condition is not met.

14. 14. The information processing apparatus according to claim 1, wherein the predetermined image processing is a synthesis process or a process for detecting an abnormality occurring in a photographed object.

15. An acquisition step of acquiring an image including a structure; a first determination step of performing a first determination as to whether or not the resolution of the image satisfies a predetermined condition; a second determination step of performing a second determination as to whether or not the degree of focus for each predetermined region of the image satisfies a predetermined condition when it is determined in the first determination that the predetermined condition is satisfied; a third determination step of performing a third determination as to whether or not the frequency components of the image satisfy a predetermined condition when it is determined in the second determination that the predetermined condition is satisfied; a fourth determination step of determining that the image is suitable for predetermined image processing if it is determined in the third determination that the image satisfies a predetermined condition, and determining that the image is not suitable for the predetermined image processing if it is determined in at least one of the first determination, the second determination, and the third determination that the image does not satisfy a predetermined condition; a display control step of controlling the display of the image and the determination result of the fourth determination step; and If it is determined in the first determination that the predetermined condition is not satisfied, the second determination and the third determination are not performed; If it is determined in the second determination that the predetermined condition is not satisfied, the third determination is not performed. An information processing method comprising:

16. A program for causing a computer to function as the information processing device according to any one of claims 1 to 14.

Images