Information processing apparatus, method of controlling information processing apparatus, and program

Abstract

To determine image quality of a captured image by taking into account regions of a target object.SOLUTION: An information processing apparatus includes: image acquisition means for acquiring a captured image including a structure to be inspected; information acquisition means for acquiring in-focus degree information indicating a degree of in-focus of each of a plurality of regions of the captured image; selection means for selecting one or more partial regions from a plurality of partial regions obtained by dividing the captured image into regions; and determination means for determining whether the captured image satisfies an in-focus condition necessary for being used in inspection, based on the in-focus information and the selected one or more partial regions.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present disclosure relates to a technique for determining the quality of a captured image. [Background technology]

[0002] Conventionally, inspections of infrastructure structures such as bridges and tunnels for abnormalities have been conducted by workers directly visually inspecting the infrastructure structures or by using tools, but in recent years, inspections have been conducted using images of infrastructure structures. In inspections using photographed images (image inspections), the photographed images must meet a certain level of image quality in order to detect abnormalities such as cracks. Patent Document 1 discloses a technology that determines whether a photographed image meets a certain level of image quality based on the degree of focus, frequency analysis results, and photographing resolution, and identifies whether the image needs to be re-photographed based on the determination result. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-75510 Summary of the Invention [Problem to be solved by the invention]

[0004] However, according to Patent Document 1, when the infrastructure structure to be inspected is included in the captured image, it may be determined that re-imaging is necessary even if the area of ​​the infrastructure structure to be inspected meets a specified image quality.

[0005] The present disclosure has been made in consideration of such problems, and has an object to provide a technique for more appropriately determining the image quality of a captured image. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, the information processing device according to the present disclosure has the following configuration. That is, the information processing device has an image acquisition means for acquiring a photographed image including a structure to be inspected, an information acquisition means for acquiring focus degree information indicating the focus degree of a plurality of regions of the photographed image, a selection means for selecting one or more partial regions from a plurality of partial regions obtained by dividing the photographed image into regions, and a determination means for determining whether the photographed image satisfies a focus condition required for use in the inspection based on the focus degree information and the selected one or more partial regions. Alternatively, the information processing device has an image acquisition means for acquiring a photographed image including a structure to be inspected, an information acquisition means for acquiring frequency information indicating a frequency analysis result of the photographed image, a selection means for selecting one or more partial regions from a plurality of partial regions obtained by dividing the photographed image into regions, and a determination means for determining whether the photographed image satisfies a focus condition required for use in the inspection based on the frequency information and the selected one or more partial regions. Effect of the Invention

[0007] According to the present disclosure, it is possible to provide a technique for more appropriately determining the image quality of a captured image. [Brief description of the drawings]

[0008] [Figure 1] FIG. 2 is a diagram illustrating a hardware configuration of an imaging system and each device. [Diagram 2] FIG. 2 is a diagram illustrating a functional configuration of an information processing device. [Diagram 3] 11 is a flowchart illustrating an operation of the information processing device. [Figure 4] FIG. 13 is a diagram illustrating an example of a determination region. [Diagram 5] 11 is a detailed flowchart of the image quality determination process (S306) (first embodiment). [Figure 6] FIG. 13 is a diagram illustrating an example of a defocus map. [Figure 7] FIG. 11 is a diagram showing an example of displaying image quality judgment results (first embodiment). [Figure 8]13 is a flowchart of image quality determination processing (S306) (second embodiment). [Figure 9] FIG. 11 is a diagram showing an example of displaying image quality judgment results (second embodiment). [Figure 10] 13 is a flowchart of an image quality determination process involving a retry. [Figure 11] 13 is a flowchart of image quality determination processing (S306) (third embodiment). [Figure 12] FIG. 13 is a diagram illustrating an example of a UI screen for setting a determination condition. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0010] (First embodiment) As a first embodiment of an information processing device according to the present disclosure, an information processing device that controls an imaging device to capture an image of a structure will be described below as an example.

[0011] <System configuration and device configuration> 1 is a diagram showing the configuration of an imaging system and the hardware configuration of each device. The imaging system is a system for inspecting (image inspection) a structure based on an image of the structure. Examples of structures include bridges, tunnels, roads, buildings, dams, embankments, and electrical equipment. The imaging system includes an information processing device 100, a photography assistant device 150, and a photography device 180, which are communicatively connected.

[0012] The information processing device 100 is a device for controlling the entire photographing process of the imaging system. The information processing device 100 has a central processing unit (CPU) 101, a read-only memory (ROM) 102, a random access memory (RAM) 103, and a hard disk drive (HDD) 104. The information processing device 100 further has a display unit 105, an operation unit 106, and a communication unit 107.

[0013] CPU 101 executes programs to perform calculations and logical decisions for various processes and controls each component connected to system bus 110. ROM 102 is a program memory that stores programs for control by CPU 101, including various processing procedures described below. RAM 103 is used as a temporary storage area such as the main memory and work area of ​​CPU 101. Note that a program memory may be realized by loading a program into RAM 103 from an external storage device connected to information processing device 100.

[0014] The HDD 104 is a storage device for storing electronic data such as image data and programs according to this embodiment. An external storage device may be used to fulfill a similar role. Here, the external storage device may be realized, for example, by a medium (recording medium) and an external storage drive for realizing access to the medium. Known examples of such media include flexible disks (FDs), CD-ROMs, DVDs, USB memories, MOs, and flash memories. The external storage device may also be a server device connected via a network.

[0015] The display unit 105 is, for example, a liquid crystal display (LCD) or an organic light emitting display (OLED), and is a device that outputs an image on a display screen. The display unit 105 may be an external device connected to the information processing device 100 by wire or wirelessly. The operation unit 106 is a device such as a keyboard or a mouse, and accepts various operations by a user.

[0016] The communication unit 107 performs wired or wireless two-way communication with other information processing devices, communication devices, external storage devices, etc., using known communication techniques. The communication unit 107 may be configured, for example, with a chip or antenna for performing wireless communication, and performs communication using public wireless such as 4G / 5G, wireless LAN, Bluetooth (registered trademark), or other wireless communication methods.

[0017] The photographing assistant device 150 is a camera platform device that can change the photographing position and the photographing direction based on the control of the information processing device 100. The photographing device 180, which will be described later, is mounted on the photographing assistant device 150. The photographing assistant device 150 also has a communication unit 151, a posture control unit 152, and a photographing instruction unit 153.

[0018] The communication unit 151 communicates with the information processing device 100 wirelessly or wiredly, and controls the position and orientation (imaging direction and position) and issues imaging instructions according to instructions from the information processing device. The communication unit 151 may be configured, for example, with a chip or antenna for wireless communication, and communicates by public wireless such as 4G / 5G, wireless LAN, Bluetooth (registered trademark), or other wireless communication method.

[0019] The attitude control unit 152 changes the attitude (imaging direction and position) of the camera platform device so that the designated imaging area of ​​the subject to be inspected can be imaged. The imaging instruction unit 153 controls the imaging device 180 set to the imaging direction and position changed by the attitude control unit 152 so as to perform imaging.

[0020] The photographing device 180 is a device that photographs based on photographing instruction information received from the information processing device 100 via the photographing auxiliary device 150. For example, the photographing device 180 has a photographing unit 181 including a photographing optical system and an imaging element. The imaging element has, for example, a full-field phase-difference imaging sensor, and also records focus degree information (defocus value) at each pixel position in the photographed image. The focus degree information will be described in detail later with reference to FIG. 8. Here, the focus degree information is data that represents a spatial (two-dimensional) distribution of defocus values ​​in the photographing range. In the following description, the data that represents the spatial distribution of defocus values ​​is also referred to as a defocus map. The defocus value is the amount of focus deviation from the distance at which the optical system of the photographing device 180 is in focus. In other words, the defocus map shows the defocus value for each pixel of the image as a spatial distribution.

[0021] Each pixel of the full-field phase-difference imaging sensor of the imaging device 180 has two photoelectric conversion units, which are referred to as divided pixel A and divided pixel B. In the full-field phase-difference imaging sensor, images A and B are output as parallax images from divided pixels A and B that are regularly arranged two-dimensionally. An image A+B obtained by adding images A and B is recorded as a recorded still image. A defocus value is calculated based on the phase difference between these parallax images. Note that, although a configuration in which the defocus value is derived on a pixel-by-pixel basis will be described as an example here, the defocus value may be derived for each predetermined region, such as a block unit having a plurality of pixels (e.g., 5 pixels x 5 pixels).

[0022] 2 is a diagram showing the functional configuration of the information processing device 100. The information processing device is a device that executes an information processing method for determining the quality of a captured image and determining the need to re-capture the image. The information processing device 100 has a control unit 200, a determination unit 201, an image acquisition unit 205, an output unit 206, and a storage unit 207. Each of these functional units is realized by the CPU 101 expanding a program stored in the ROM 102 into the RAM 103 and executing it. However, some or all of these functional units may be configured to be realized by hardware.

[0023] The control unit 200 controls the execution of various processes by each functional unit. The judgment unit 201 is a functional unit that judges whether or not an image satisfies a predetermined image quality. The judgment unit 201 includes an image information acquisition unit 202, a judgment condition setting unit 203, and an image information judgment unit 204. The image information acquisition unit 202 analyzes an image and acquires the degree of focus and frequency information of the image. The judgment condition setting unit 203 sets judgment conditions, which will be described later with reference to FIG. 5. The image information judgment unit 204 judges the image quality based on the judgment conditions set by the judgment condition setting unit 203, based on the image information acquired by the image information acquisition unit 202. The image acquisition unit 205 acquires a captured image obtained by imaging by the imaging device 180. The output unit 206 outputs the captured image and the result of judgment by the judgment unit 201. The storage unit 207 holds the captured image, the judgment conditions set by the judgment condition setting unit 203, and the judgment result obtained by the image information judgment unit 204.

[0024] <Device Operation> 3 is a flowchart illustrating the operation of the information processing device 100. The flowchart starts when the information processing device 100 executes a program.

[0025] In S301, the control unit 200 performs a photographing process. The photographing process is a process in which the information processing device 100 controls the photographing assistant device 150 and the photographing device 180 to perform photographing.

[0026] In S302, the image acquisition unit 205 acquires a captured image. Specifically, one or more captured images to be processed in the following steps S304 to S306 are acquired.

[0027] In S303, the judgment condition setting unit 203 acquires the judgment conditions designated by the user via the operation unit 106, and stores them in the storage unit 207. The stored judgment conditions are used in the image quality judgment process in S306, which will be described later.

[0028] In S304 to S306, a process is performed to determine whether or not each of the one or more captured images acquired in S302 satisfies the shooting conditions and image quality.

[0029] In S304, the image information acquisition unit 202 acquires image information of the input image and stores it in the storage unit 207. Then, the image information determination unit 204 performs a shooting condition determination process for determining shooting condition information when the captured image was captured, from the acquired image information.

[0030] In S305, the image information determination unit 204 determines whether to proceed to processing in S306 for the image to be processed based on the determination result of S304. If it is determined that the captured image matches the specified shooting conditions (Yes), the process proceeds to S306, and if it is determined that it does not match (No), the determination information "NG" indicating that it cannot be used is stored in the storage unit 207 and the process proceeds to S304. Note that instead of storing the determination information "NG", "not matching (the determination conditions)" may be stored. When proceeding to S304, an unprocessed captured image to be the next processing target is selected.

[0031] In the photographing condition determination process (S304), if the photographing conditions of the determined photographed image do not match the predetermined photographing conditions, the photographed image currently being processed is determined to be an image that requires re-shooting, and is stored in the storage unit 207. Also, by performing the determination process of S304 using the photographing conditions, it is possible to determine a photographed image that requires re-shooting without performing the image quality determination process of S306 described later. Also, by performing the photographing condition determination process of S304, which has a lower load than the image quality determination process of S306 described later, at an earlier stage, it is possible to shorten the processing time.

[0032] In S306, the image information determination unit 204 performs an image quality determination process for determining the image quality of the captured image. Here, image quality means the degree of blurring or shaking in the captured image (the smaller the degree of blurring or shaking, the higher the image quality). Details of the image quality determination process will be described later with reference to the flowchart in FIG. 5. In the image quality determination process in S306, the image quality is determined using the determination conditions set in S303, for example, using focus level information indicating the focus level of the captured image, and the determination information ("OK" or "NG") is stored in the storage unit 207. The determination conditions in the image quality determination using focus level information include, for example, a focus level threshold value for each determination area shown in FIG. 4.

[0033] The focus degree information is used because images that are not properly focused on the inspection surface are likely to prevent proper image processing required for the inspection, such as abnormality detection and synthesis, and should not be used for the inspection. A shift in the focus position can occur, for example, when a gust of wind occurs during the shooting process, causing the camera 180 to move. Also, a shift in the focus position can occur when an obstruction (such as another worker or a bird) gets between the camera and the inspection surface and the camera enters the range-finding point of the camera, causing autofocus control to be performed on the obstruction.

[0034] In this example, in the image quality determination process (S306), image quality is determined using focus level information. However, image quality may be determined based on the resolution indicated by the number of pixels per unit length of the inspection surface.

[0035] FIG. 4 is a diagram showing an example of a judgment area. The judgment area is, for example, an area in the captured image where an infrastructure structure to be inspected is captured. More specifically, it is an image area in the captured image that is used during inspection (image inspection). FIG. 4(a) and FIG. 4(b) respectively show examples of judgment areas in the image quality judgment process of S306. An outer frame 401 and an outer frame 404 are frames corresponding to the image size of the input image (captured image). The input image is divided into a plurality of areas, and an area 402 designated by the user from among the divided plurality of areas is set as a judgment area, and an area 403 not designated by the user is set as a non-judgment area. The remaining seven areas are similarly designated by the user, and as a result, judgment areas (six partial areas) shown as hatched areas and non-judgment areas (three partial areas) shown as white areas are set.

[0036] In the example of FIG. 4(a), the shape of each divided region is rectangular, but other shapes may be used. For example, a circular shape or a freehand drawn shape may be used. In addition, although an example of selecting a judgment region from a region divided into 9 equal parts (3×3) is shown, other shapes may be used. For example, the region may be divided into a different number of parts, such as 15 equal parts (5×3) or 24 equal parts (6×4). In addition, as shown in FIG. 4(b), a configuration may be used in which a region to be judged can be selected in detail from regions 405 obtained by further dividing the divided region 402.

[0037] In S307, the image information determination unit 204 determines whether or not the processes of S304 to S306 have been completed for all captured images input in S302. If the processes have been completed for all captured images (Yes), the process proceeds to S308, and if not (No), the process proceeds to S304. If the process proceeds to S304, an unprocessed captured image to be the next processing target is selected.

[0038] In S308, the image information determination unit 204 determines whether or not the image quality determination result is "NG" in the determination information stored in the storage unit 207. If it is included (Yes), the process proceeds to S309, and if it is not included (No), the process ends.

[0039] In S309, the image information determination unit 204 identifies captured images that require re-shooting, and the output unit 206 outputs information on the identified captured images. In the first embodiment, the output unit 206 outputs information indicating the determination result in S306 in association with each of the captured images that were the subject of the determination. For example, the output unit 206 outputs information indicating that the image quality satisfies the conditions, such as "OK", or information indicating that the image quality does not satisfy the conditions. In addition, the output unit 206 may be configured to output information indicating that the image quality can be used for image processing such as abnormality detection and synthesis, or information indicating that re-shooting is required or not required.

[0040] Furthermore, the output of S309 may be configured so long as it is possible to identify the captured images that have been identified as needing to be re-taken, and may be configured to store the captured images that need to be re-taken in a specified folder (e.g., an "NG" folder). Also, a composite image (such as FIG. 7 described later) may be created based on the capture position / range of each captured image to present the worker with the capture position / range that needs to be re-taken. Also, a screen display or sound may be output to indicate that the captured images do not satisfy the image quality requirements or that need to be re-taken are included.

[0041] FIG. 5 is a detailed flowchart of the image quality determination process (S306) in the first embodiment.

[0042] In S501, the image information determination unit 204 acquires focus degree information included in the image to be processed. Here, the focus degree information is a defocus map (information in which a defocus value for each region of a captured image is recorded) that is information acquired together with an image captured by an imaging device having a full-field phase-difference imaging sensor. The focus degree information also includes accuracy (accuracy of focus degree determination) that indicates the certainty of the defocus value recorded for each region of the image.

[0043] In S502, the image information determination unit 204 calculates the area (for example, the total number of pixels in the determination area) of the determination area (six shaded areas in FIG. 4(a)) designated by the user. Note that the area calculated here includes areas where the accuracy of the focus degree determination is determined to be low.

[0044] In S503, the image information determination unit 204 uses the focus level information acquired in S501 to calculate the "area of ​​the in-focus area (focused area)" in the determination area (area 402). For example, the area of ​​the area where the defocus value is less than a predetermined value is calculated. For example, the number of pixels where the defocus value is "0" in FIG. 6 is calculated.

[0045] Fig. 6 is a diagram showing an example of defocus values, which are focus level information. Fig. 6(a) shows a contour map (contour diagram) of two-stage (1-bit) defocus values ​​("0", "1"), and Fig. 6(b) shows a contour map of four-stage (2-bit) defocus values ​​("0", "1", "2", "3"). However, the number of stages (number of bits) of the defocus values ​​is not limited to these.

[0046] Outer frames 601 and 611 are frames corresponding to the image size of the input image (captured image). Numerical values ​​602 and 612 are defocus values ​​in the respective regions. Lines 603 and 613 indicate the boundary between regions with different defocus values. The defocus value is a numerical value obtained by digitizing the amount of shift in front of and behind the focus of an object as an absolute value. Here, it is assumed that the defocus value (defocus map) is obtained from a full-field phase-difference imaging sensor, but it may also be obtained using parallax images captured by a stereo camera or the like.

[0047] Note that the defocus value may be expressed as a positive value (forward focus deviation) and a negative value (backward focus deviation) instead of an absolute value. Also, for an area where the accuracy of the defocus value is determined to be low, the maximum defocus value that can be set may be recorded regardless of the recorded defocus value.

[0048] 6, an area with a defocus value of "0" is an area that is in focus (an in-focus area) in terms of the focusing unit of the image capture device 150. An area with a defocus value greater than "0" is an area that is out of focus (an unfocused area). A larger defocus value indicates a larger amount of focus deviation (poor focus).

[0049] In S504, the image information determination unit 204 determines whether the ratio of the areas (number of pixels) calculated in S502 and S503 is equal to or greater than a predetermined threshold. For example, referring to FIG. 4(a) and FIG. 6(a), the calculation is performed with the area (number of pixels) of the determination area 402 as the denominator and the number of pixels in the determination area 402 with a defocus value of "0" as the numerator. That is, the ratio of the in-focus area (area ratio) in the determination area 402 is calculated. If the ratio is equal to or greater than the threshold (Yes), the process proceeds to S505, and if it is less than the threshold (No), the process proceeds to S506. For example, if the threshold is "80%, the determination area 402 is determined to be equal to or greater than the threshold in FIG. 6(a), and is determined to be less than the threshold in FIG. 6(b). The threshold can be set arbitrarily by the user. The threshold may be configured to be determined based on the relationship between the ratio of the in-focus area and the detection accuracy for detecting anomalies. For example, a threshold value may be set in advance as a proportion of the in-focus area such that the accuracy of detecting anomalies is 95% or higher.

[0050] In S505, the image information determination unit 204 determines that the captured image to be processed is of "image quality OK." That is, the captured image satisfies the focusing conditions required for use in the inspection, and therefore does not need to be re-captured. On the other hand, in S506, the image information determination unit 204 determines that the captured image to be processed is of "image quality NG." That is, it determines that re-capture is required. In S505 and S506, the determination results are recorded in the recording unit 207.

[0051] In S507, the image information determination unit 204 generates an image on which the determination result is superimposed, using information on the determination result. Then, the output unit 206 performs display control to display the generated image on the display unit 105. The generated image will be described with reference to FIG.

[0052] Fig. 7 is a diagram showing an example of displaying the image quality judgment result in the first embodiment. Here, an example is shown in which the contour map shown in Fig. 6(b) is used. A result image 700 includes divided regions 701 (six regions indicated by diagonal lines) used for judgment, divided regions 702 (three regions indicated by white lines) not used for judgment, and a frame region 707.

[0053] Also, in the resultant image 700, there are shown an area 703 with a defocus value of "0," an area 704 with a defocus value of "1," an area 705 with a defocus value of "2," and an area 706 with a defocus value of "3." Here, the defocus value is superimposed only on the divided area 701 to be determined.

[0054] Furthermore, in the frame region 707, a focus level determination icon 708 is displayed as an indication of the determination result. The result image 700 shows an example in which the focus level determination (indicated as "F" which means focus) is OK. For example, the display form of "F" may be different (e.g., black / white text) depending on whether the determination is OK or NG. Alternatively, the color of the icon or the color of the frame region 707 may be changed (e.g., blue / red). This allows the user to more easily understand the determination result.

[0055] In S503, the area of ​​the region where the defocus value is "0" is calculated, but this is not limiting. For example, the area of ​​the region where the defocus value is equal to or less than a threshold value, for example, equal to or less than 1 (i.e., "0" or "1"), may be calculated.

[0056] Also, in S503, the areas of the regions where the defocus value is "3" may be obtained, and if the ratio of the areas is less than a threshold, the process may proceed to S505, and if the ratio of the areas exceeds the threshold, the process may proceed to S506. Furthermore, in S503, a first area of ​​the region where the defocus value is "0" and a second area of ​​the region where the defocus value is "3" may be obtained. In this case, if the ratio of the first areas exceeds the first threshold and the ratio of the second areas is less than the second threshold, the process may proceed to S505, and otherwise the process may proceed to S506.

[0057] In this way, it is possible to determine whether the degree of focus of the image satisfies each of two or more conditions, and to determine whether the image quality of the image is OK (can be used for image processing such as abnormality detection and composition). It may also be configured to be used to determine whether re-imaging is necessary.

[0058] In the above description, an example has been described in which the degree of focus in the determination area designated by the user is used in the image quality determination process (S306). However, the image quality may be determined by other methods. For example, the degree of blur may be calculated from the distribution amount of high frequency components and low frequency components using frequency analysis processing, and the image quality may be determined based on a predetermined threshold value of the intensity of blur. That is, blur information indicating the degree of blur in the determination area may be generated using frequency analysis processing, and based on the blur information, it may be determined whether the image quality of the image is OK or not. Furthermore, the determination may be made by combining the information indicating the degree of blur and the information on the degree of focus.

[0059] As described above, according to the first embodiment, when determining the image quality of a captured image used for image inspection of an infrastructure structure, the image quality is determined for a determination area that takes into account the area in which the infrastructure structure to be inspected is captured. This makes it possible to reduce the number of cases where the image quality is determined as NG due to parts other than the infrastructure structure to be inspected in the captured image, requiring re-imaging.

[0060] Second embodiment Subjects without any contrast between light and dark (such as flat, monochromatic subjects) are considered to be subjects that are difficult to focus on, and infrastructure structures made of concrete are examples of this. Therefore, in an image of an infrastructure structure made of concrete, there may be areas where the accuracy of the judgment using the focus level information is low. When an area with low accuracy exists, the deviation width (variance) of the defocus value in that area becomes wide, and the image quality judgment result becomes unreliable.

[0061] Therefore, in the second embodiment, a form in which image quality judgment is performed only in an area where the accuracy of judgment (focus degree judgment) using focus degree information is equal to or higher than a predetermined level will be described. Also, a form in which focus degree judgment is retried while changing the judgment area to reduce the burden on the user when the focus degree judgment is NG will be described. Hereinafter, the same parts as in the first embodiment will be omitted, and only the differences from the first embodiment will be described.

[0062] <Device Operation> Fig. 8 is a flowchart of the image quality determination process (S306) in the second embodiment. In the first embodiment (Fig. 5), an example is shown in which only areas where the accuracy of the determination using the focus level information is higher than a predetermined condition are used. The difference is that S502, S503, S504, and S507 in Fig. 5 are replaced with S801, S802, S803, and S804.

[0063] In S801, the image information determination unit 204 calculates the area of ​​an area with high accuracy of focus determination within a determination area (six shaded areas in FIG. 4(a)) designated by a user. For example, the number of pixels in the determination area where the accuracy (e.g., variance) of the focus determination recorded for each partial area (each pixel) is less than a predetermined value is calculated.

[0064] In S802, the image information determination unit 204 uses the focus level information acquired in S501 to calculate the "area of ​​the in-focus area" in the area where the focus level determination accuracy is high determined in S801. For example, the area of ​​the area where the defocus value is less than a predetermined value is calculated. For example, the number of pixels where the defocus value is "0" in FIG. 6 is calculated.

[0065] In S803, the image information determination unit 204 determines whether the ratio of the areas (number of pixels) calculated in S801 and S802 (the area in S801 is the denominator and the area in S802 is the numerator) is equal to or greater than a predetermined threshold. If the ratio is equal to or greater than the threshold (Yes), the process proceeds to S505, and if it is less than the threshold (No), the process proceeds to S506.

[0066] In S804, the image information determination unit 204 uses the information on the determination result to generate an image on which the determination result is superimposed, and outputs the image via the output unit 206. The generated image will be described with reference to FIG.

[0067] FIG. 9 is a diagram showing a display example of the image quality judgment result in the second embodiment. Here, an example is shown in which the contour map shown in FIG. 6(b) is used, as in the first embodiment (FIG. 7). The result image 900 includes divided regions 901 (four regions indicated by diagonal lines and a white background) used for judgment, and divided regions 902 (two regions indicated by diagonal lines and a gray background) in which judgment was performed using focus level information, but the area of ​​the region with high accuracy of the judgment is less than a predetermined threshold. It also includes divided regions 702 (three regions indicated by white) not used for judgment, and a frame region 707. The divided regions 902 include many regions with low accuracy of the focus level judgment. By presenting the divided regions 902, the user can know that a closer examination is required as a judgment result.

[0068] Also, in the resultant image 900, there are shown an area 703 with a defocus value of "0," an area 704 with a defocus value of "1," an area 705 with a defocus value of "2," and an area 706 with a defocus value of "3." Here, the defocus values ​​are superimposed and displayed only for the divided areas 901 and 902.

[0069] Here, as an indication that the area of ​​the region with high focus level determination accuracy is less than the threshold, divided region 902 is expressed in a form different from divided region 901. However, it may be expressed by making the frame of the region thicker or changing the color, etc. This allows the user to easily confirm that there is a region with low focus level determination accuracy in the captured image and that this region was not used in the determination.

[0070] Fig. 10 is a flowchart of image quality judgment processing involving retry. Specifically, focus level judgment (image quality judgment) is performed on the entire captured image, and if the image quality is judged to be NG, image quality judgment is performed on each partial area divided into the captured image area. Then, if the number of partial areas with OK image quality is not a predetermined number or more, image quality judgment is performed on areas in each partial area where the accuracy of focus level judgment is a predetermined value or more. The following describes the parts that are different from Figs. 5 and 8.

[0071] In S1001, the image information determination unit 204 calculates the area of ​​the entire captured image (for example, the total number of pixels). In S1002, the image information determination unit 204 calculates the "area of ​​the in-focus region" in the captured image (entire region) using the focus level information acquired in S501.

[0072] In S1003, the image information determination unit 204 determines whether the ratio of the areas (number of pixels) calculated in S1001 and S1002 (the area in S1001 is the denominator and the area in S1002 is the numerator) is equal to or greater than a predetermined threshold. If the ratio is equal to or greater than the threshold (Yes), the process proceeds to S505, and if it is less than the threshold (No), the process proceeds to S1004.

[0073] In S1004, the image information determination unit 204 determines one of partial regions obtained by dividing the captured image by a predetermined number of divisions as a determination region, and calculates the area of ​​the determination region.

[0074] In S1005, the image information determination unit 204 calculates the "area of ​​the in-focus region" within the determination region using the focus level information acquired in S501.

[0075] In S1006, the image information determination unit 204 determines whether the ratio of the areas (number of pixels) calculated in S1004 and S1005 (the area in S1004 is the denominator and the area in S1005 is the numerator) is equal to or greater than a predetermined threshold. If the ratio is equal to or greater than the threshold (Yes), the process proceeds to S1007, and if it is less than the threshold (No), the process proceeds to S1008.

[0076] In S1007, the image information determination unit 204 counts the number of areas where the ratio is determined to be equal to or greater than the predetermined threshold value (Yes) in S1006, and records the number in the recording unit 207.

[0077] In S1008, the image information determination unit 204 determines whether the processes of S1004 to S1006 have been completed for all partial regions divided by a predetermined number of divisions. If the processes have been completed for all partial regions (Yes), the process proceeds to S1009, and if not (No), the process proceeds to S1004. When proceeding to S1004, an unprocessed partial region to be the next processing target is selected.

[0078] In S1009, the image information determination unit 204 determines whether the number of regions counted in S1007 is equal to or greater than a predetermined threshold. If it is equal to or greater than the predetermined threshold (Yes), the process proceeds to S505. If it is less than the threshold (No), the process proceeds to S1010.

[0079] In S1010, the image information determination unit 204 divides the captured image into a predetermined number of partial regions, determines one of the partial regions as a determination region, and calculates the area of ​​the region in the determination region where the focus degree determination is highly accurate. In S1011, the image information determination unit 204 calculates the "area of ​​the in-focus region" in the region calculated in S1010, using the focus degree information acquired in S501.

[0080] In S1012, the image information determination unit 204 determines whether the ratio of the areas (number of pixels) calculated in S1010 and S1011 (the area in S1010 is the denominator and the area in S1011 is the numerator) is equal to or greater than a predetermined threshold. If the ratio is equal to or greater than the threshold (Yes), the process proceeds to S1013, and if it is less than the threshold (No), the process proceeds to S1014.

[0081] In S1013, the image information determination unit 204 counts the number of areas where the ratio is determined to be equal to or greater than the predetermined threshold value (Yes) in S1012, and records the number in the recording unit 207.

[0082] In S1014, image information determination unit 204 determines whether or not the processes of S1010 to S1012 have been completed for all partial regions divided by a predetermined number of divisions. If the processes have been completed for all partial regions (Yes), the process proceeds to S1015, and if not (No), the process proceeds to S1010. When proceeding to S1010, an unprocessed partial region to be the next processing target is selected.

[0083] In S1015, the image information determination unit 204 determines whether the number of regions counted in S1013 is equal to or greater than a predetermined threshold. If it is equal to or greater than the predetermined threshold (Yes), the process proceeds to S505. If it is less than the threshold (No), the process proceeds to S506.

[0084] As described above, according to the second embodiment, image quality is judged only in areas where the accuracy of the judgment using the focus level information (focus level judgment) is equal to or higher than a predetermined level. This makes it possible to reduce the number of judgment areas where the image quality is judged to be NG, and thus reduce the need for re-shooting.

[0085] In addition, the focus determination is retried while changing the determination area (whole area → each partial area → each area with high focus determination accuracy within the partial area). This makes it possible to reduce the burden on the user when the focus determination is NG.

[0086] Third embodiment In the first and second embodiments, the image quality is determined using focus level information (defocus map). On the other hand, as described above, it is also possible to determine the image quality using the degree of blur based on frequency analysis processing. Therefore, in the third embodiment, a determination is made using frequency information of the captured image in addition to the determination using focus level information of the captured image.

[0087] The frequency information is obtained by performing frequency analysis processing on the captured image and calculating the average value of the obtained frequency components. If the calculated average value is small, it indicates that there are few high frequency components (i.e., parts with sharp edges) and the degree of blur or shaking is strong. In addition to blur determination using focus level information, it is also possible to perform blur determination using frequency analysis.

[0088] <Device Operation> FIG. 11 is a flowchart of the image quality judgment process (S306) in the third embodiment. This embodiment differs from the first embodiment (FIG. 5) in that judgment using frequency information (S1101 to S1103) is also performed. In S1101, the image information determination unit 204 determines whether or not to perform a determination using frequency information for the determination area in which the ratio is determined to be equal to or greater than the predetermined threshold in S504. If the determination is to be performed (Yes), the process proceeds to S1102, and if the determination is not to be performed, the process proceeds to S505. Whether or not to perform the process is specified in advance by the user via a setting screen 1201, which will be described with reference to FIG.

[0089] In S1102, the image information determination unit 204 performs frequency analysis processing on the determination area (where the ratio is determined to be equal to or greater than the predetermined threshold in S504) to obtain frequency component values. For example, the horizontal and vertical frequency components are calculated using a wavelet transform. The average value of the obtained frequency components is calculated, and this calculated value is set as the frequency component value.

[0090] In S1103, the image information determination unit 204 determines whether the frequency component value calculated in S1102 is equal to or greater than a predetermined threshold. If it is equal to or greater than the threshold (Yes), the process proceeds to S505, and if it is less than the threshold (No), the process proceeds to S506. That is, if it is equal to or greater than the threshold, the determination unit 204 determines that the determination area being watched is an image with little blur (an image that satisfies the blur conditions required for use in inspection). On the other hand, if it is less than the threshold, the determination unit 204 determines that the determination area being watched is an image that is heavily influenced by blur or blur.

[0091] In this way, by performing blur determination processing using frequency analysis after focus determination, it is possible to determine the occurrence of blur due to blur, and more appropriately determine whether the captured image is suitable as an inspection image. An example of a situation in which blur occurs even when the image is in focus is a situation in which a camera is mounted on an aircraft such as a drone to take a picture. In this case, the drone may shake due to the influence of wind during shooting, which may cause blur.

[0092] <Settings screen> 12 is a diagram showing a setting screen 1201 which is an example of a GUI screen for setting judgment conditions. The setting screen 1201 is a setting screen for performing judgment pattern setting 1202. The judgment pattern setting 1202 includes three sections: a judgment retry setting section 1203, a focus level judgment setting section 1204, and a blur judgment setting section 1205. Check boxes 1203b, 1204b, and 1205b and input areas 1206, 1207, and 1208 are arranged in the judgment retry setting section 1203, the focus level judgment setting section 1204, and the blur judgment setting section 1205.

[0093] Check boxes 1203b, 1204b, and 1205b are user interface (UI) components for specifying whether or not to perform a determination retry, a focus level determination, or a blur determination, respectively. Input areas 1206, 1207, and 1208 are UI components for specifying a determination threshold value for the determination retry, the focus level determination, and the blur determination, respectively.

[0094] For example, a setting screen 1201 shown in Fig. 12 shows a state in which determination retry is set to OFF, focus level determination is set to ON, and blur determination is set to ON. Therefore, the focus level determination retry shown in Fig. 10 is not performed, and the focus level determination and blur determination shown in Fig. 11 are performed. Therefore, while input fields 1207 and 1208 are in a state in which input is possible, input field 1206 is grayed out and cannot be input.

[0095] The input area 1207a is an input area for specifying a predetermined value (defocus value threshold) in S503. The input area 1207b is an input area for specifying a ratio threshold in S504. The input area 1207c is a UI widget for setting an area for calculating the area in S502 and S503. For example, each area is shown as nine divided areas configured as a button, and the area that receives a button press by the user becomes the determination area.

[0096] By specifying the area of ​​the infrastructure in the captured image as the input area 1207c, it becomes possible to more appropriately judge the image quality of the infrastructure. In addition, by making the area to be judged smaller (reducing the area), the amount of calculation processing is reduced, and speed improvement is expected.

[0097] The input area 1208a is an input area for specifying the threshold value of the frequency component value in S1103. The input area 1208b is a UI part for setting the area for calculating the frequency component value in S1102. For example, it is shown as nine divided areas each configured as a button, and the area that receives the button press by the user becomes the determination area.

[0098] By specifying the area of ​​the infrastructure in the captured image in the input area 1208b, the same effect as that in the input area 1207c can be obtained. In addition, when the structure to be inspected is a bridge pier, it is possible to suppress the inclusion of high-frequency components caused by plants or the like being reflected on the back side of the structure.

[0099] As described above, according to the third embodiment, in addition to the determination based on the focus level information of the captured image, a blur determination is also performed using the frequency information of the captured image. This makes it possible to more appropriately determine the influence of blur that may occur when shooting using a drone. In the above description, the focus level determination is performed before the blur determination, but the order may be reversed. Also, the two determinations may be performed in parallel, and then the results may be integrated.

[0100] The disclosure of this specification includes the following information processing device, control method, and program. (Item 1) An image acquisition means for acquiring a photographed image including a structure to be inspected; an information acquisition means for acquiring focus degree information indicating a focus degree of a plurality of regions of the photographed image; A selection means for selecting one or more partial regions from a plurality of partial regions obtained by dividing the captured image into regions; a determination means for determining whether the captured image satisfies a focus condition required for use in an inspection, based on the focus degree information and the one or more selected partial regions; 13. An information processing device comprising: (Item 2) The determination means is Identifying an area of ​​a focused region of the one or more partial regions based on the focus degree information; Based on a ratio of an area of ​​the focused region to an area of ​​the one or more partial regions, it is determined whether the captured image satisfies a focusing condition required for use in the inspection. 2. The information processing device according to item 1, (Item 3) The determining means determines that the captured image satisfies a focusing condition required for use in an inspection when the ratio is equal to or greater than a threshold value. 3. The information processing device according to item 2. (Item 4) The determination means is Identifying, as the focused region, a first region among the one or more partial regions, in which the accuracy of determination using the focus degree information is equal to or higher than a predetermined level; The ratio of the area of ​​the focused area to the area of ​​the first area is calculated as the ratio. 4. The information processing device according to item 2 or 3. (Item 5) The one or more partial regions are regions in which the structure appears in the captured image. 5. The information processing device according to any one of items 1 to 4. (Item 6) The selection means includes a reception means for receiving a selection of the one or more partial regions from a user. 6. The information processing device according to any one of items 1 to 5. (Item 7) The information acquisition means further acquires shooting condition information indicating shooting conditions when the photographed image was photographed, The determination means determines that the captured image cannot be used for inspection if the photographing conditions do not match predetermined photographing conditions. 7. The information processing device according to any one of items 1 to 6. (Item 8) The image capturing device further includes a display control device for displaying the captured image and the determination result by the determination device on the captured image on a display unit in association with each other. 8. The information processing device according to any one of items 1 to 7, (Item 9) the focus level information is a defocus map corresponding to the captured image, The display control means displays a contour map of the defocus map superimposed on the captured image displayed on the display unit. 9. The information processing device according to item 8, (Item 10) The information acquisition means further acquires frequency information indicating a result of an analysis of a frequency of the photographed image, The determination means derives frequency components corresponding to the one or more partial regions based on the frequency information for the captured image determined to satisfy the focusing condition. 10. The information processing device according to any one of items 1 to 9. (Item 11) The determining means determines that the captured image satisfies a blur condition required for use in an inspection when the frequency component is equal to or greater than a threshold value. 11. The information processing device according to item 10, (Item 12) An image acquisition means for acquiring a photographed image including a structure to be inspected; an information acquisition means for acquiring frequency information indicating a result of an analysis of the frequency of the photographed image; A selection means for selecting one or more partial regions from a plurality of partial regions obtained by dividing the captured image into regions; a determination means for determining whether the captured image satisfies a focusing condition required for use in an inspection, based on the frequency information and the one or more selected partial regions; 13. An information processing device comprising: (Item 13) The determination means is Identifying a first region in which no blur or shaking occurs in the one or more selected partial regions based on the frequency information; Based on a ratio of an area of ​​the first region to an area of ​​the one or more partial regions, it is determined whether the captured image satisfies a condition required for use in an inspection. 13. The information processing device according to item 12, (Item 14) An image acquisition step of acquiring a photographed image including a structure to be inspected; an information acquiring step of acquiring focus degree information indicating a focus degree of a plurality of regions of the captured image; a selection step of selecting one or more partial regions from a plurality of partial regions obtained by dividing the captured image into regions; a determination step of determining whether the captured image satisfies a focus condition required for use in an inspection based on the focus degree information and the one or more selected partial regions; 23. A method for controlling an information processing apparatus comprising: (Item 15) An image acquisition step of acquiring a photographed image including a structure to be inspected; an information acquisition step of acquiring frequency information indicating a result of an analysis of the frequency of the photographed image; a selection step of selecting one or more partial regions from a plurality of partial regions obtained by dividing the captured image into regions; a determination step of determining whether the captured image satisfies a focusing condition required for use in an inspection based on the frequency information and the one or more selected partial regions; 23. A method for controlling an information processing apparatus comprising: (Item 16) 16. A program for causing a computer to execute the control method according to item 14 or 15.

[0101] (Other embodiments) The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions.

[0102] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0103] 200 control unit; 201 judgment unit; 202 image information acquisition unit; 203 judgment condition setting unit; 204 image information judgment unit; 205 image acquisition unit; 206 output unit; 207 storage unit

Claims

1. An image acquisition means for acquiring a photographed image including a structure to be inspected; an information acquisition means for acquiring focus degree information indicating a focus degree of a plurality of regions of the photographed image; a selection means for selecting one or more partial regions from a plurality of partial regions obtained by dividing the captured image into regions; a determination means for determining whether the captured image satisfies a focusing condition required for use in an inspection, based on the focus degree information and the one or more selected partial regions; 13. An information processing device comprising:

2. The determination means is Identifying an area of ​​a focused region of the one or more partial regions based on the focus degree information; Based on a ratio of an area of ​​the focused region to an area of ​​the one or more partial regions, it is determined whether the captured image satisfies a focusing condition required for use in the inspection.

2. The information processing apparatus according to claim 1,

3. The determining means determines that the captured image satisfies a focusing condition required for use in an inspection when the ratio is equal to or greater than a threshold value.

3. The information processing apparatus according to claim 2.

4. The determination means is Identifying, as the focused region, a first region among the one or more partial regions, in which the accuracy of determination using the focus degree information is equal to or higher than a predetermined level; A ratio of an area of ​​the focused region to an area of ​​the first region is calculated as the ratio.

3. The information processing apparatus according to claim 2.

5. The one or more partial regions are regions in which the structure appears in the captured image.

2. The information processing apparatus according to claim 1,

6. The selection means includes a reception means for receiving a selection of the one or more partial regions from a user.

2. The information processing apparatus according to claim 1,

7. The information acquisition means further acquires shooting condition information indicating shooting conditions when the photographed image was photographed, The determination means determines that the captured image cannot be used for inspection if the photographing conditions do not match predetermined photographing conditions.

2. The information processing apparatus according to claim 1,

8. The image capturing device further includes a display control device for displaying the captured image and the determination result by the determination device on the captured image on a display unit in association with each other.

2. The information processing apparatus according to claim 1,

9. the focus level information is a defocus map corresponding to the captured image, The display control means displays a contour map of the defocus map superimposed on the captured image displayed on the display unit.

9. The information processing apparatus according to claim 8,

10. The information acquisition means further acquires frequency information indicating a result of an analysis of a frequency of the photographed image, The determination means derives frequency components corresponding to the one or more partial regions based on the frequency information for the captured image determined to satisfy the focusing condition.

2. The information processing apparatus according to claim 1,

11. The determining means determines that the captured image satisfies a blur condition required for use in an inspection when the frequency component is equal to or greater than a threshold value.

11. The information processing apparatus according to claim 10,

12. An image acquisition means for acquiring a photographed image including a structure to be inspected; an information acquisition means for acquiring frequency information indicating a result of an analysis of the frequency of the photographed image; a selection means for selecting one or more partial regions from a plurality of partial regions obtained by dividing the captured image into regions; a determination means for determining whether the captured image satisfies a focusing condition required for use in an inspection, based on the frequency information and the one or more selected partial regions; 13. An information processing device comprising:

13. The determination means is Identifying a first region in which no blur or shaking occurs in the one or more selected partial regions based on the frequency information; Whether the captured image satisfies a condition required for use in an inspection is determined based on a ratio of an area of ​​the first region to an area of ​​the one or more partial regions.

13. The information processing apparatus according to claim 12.

14. An image acquisition step of acquiring a photographed image including a structure to be inspected; an information acquiring step of acquiring focus degree information indicating a focus degree of a plurality of regions of the captured image; a selection step of selecting one or more partial regions from a plurality of partial regions obtained by dividing the captured image into regions; a determination step of determining whether the captured image satisfies a focus condition required for use in an inspection based on the focus degree information and the one or more selected partial regions; 23. A method for controlling an information processing apparatus comprising:

15. An image acquisition step of acquiring a photographed image including a structure to be inspected; an information acquisition step of acquiring frequency information indicating a result of an analysis of the frequency of the photographed image; a selection step of selecting one or more partial regions from a plurality of partial regions obtained by dividing the captured image into regions; a determination step of determining whether the captured image satisfies a focusing condition required for use in an inspection, based on the frequency information and the one or more selected partial regions; 23. A method for controlling an information processing apparatus comprising:

16. A program for causing a computer to execute the control method according to claim 14 or 15.

Images