Information processing method, information processing apparatus, and program

Abstract

To provide an information processing method that can accurately determine a blur in a photographed image.SOLUTION: An information processing method includes: a conversion step (S701) of converting a photographed image into first frequency characteristic data; a generation step (S704) of generating second frequency characteristic data by removing data of a predetermined area from the first frequency characteristic data; and a determination step (S705-S708) of determining whether the photographed image is blurred on the basis of the second frequency characteristic data. In the generation step, the predetermined area is determined according to whether the predetermined area falls within a predetermined range with reference to statistics obtained on the basis of the first frequency characteristic data.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to an information processing method, an information processing device, and a program. [Background technology]

[0002] In the inspection of buildings, when detecting abnormalities such as cracks from captured images, the captured images must be in focus, must not be blurred, and must be clear. In addition, to detect small abnormalities within the inspection target area, high-resolution images are required, so in the case of images used in the inspection of large-scale structures, multiple high-resolution images may be synthesized to generate a single composite image. Patent Document 1 discloses a method for determining abnormalities such as defects in image files used for synthesis, and for identifying images that need to be re-shot depending on the presence or absence of abnormalities. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-152737 A Summary of the Invention [Problem to be solved by the invention]

[0004] In the method disclosed in Patent Document 1, even if the image quality of the captured image is poor, such as an image that was not properly focused when captured, an image that was blurred when captured, or an image with low resolution, it is not determined that re-capture is necessary as long as the image file is normal.

[0005] The blur of a captured image can be determined by analyzing the frequency characteristics of the captured image. However, if the captured image contains an area that is not the subject of inspection, the accuracy of the determination may be reduced due to the frequency characteristics of that area.

[0006] SUMMARY OF THE PRESENT DISCLOSURE In view of the above, an object of the present invention is to provide an information processing method capable of determining the blur of a captured image with high accuracy. [Means for solving the problem]

[0007] An information processing device as one aspect of the present invention includes a conversion step of converting a captured image into first frequency characteristic data, a generation step of generating second frequency characteristic data by excluding data of a predetermined region from the first frequency characteristic data, and a determination step of determining whether or not the captured image is blurred based on the second frequency characteristic data, wherein in the generation step, the predetermined region is determined depending on whether or not it belongs to a predetermined range based on a statistical quantity obtained based on the first frequency characteristic data.

[0008] Other objects and features of the present invention are illustrated in the following examples. Effect of the Invention

[0009] According to the present invention, it is possible to provide an information processing method capable of determining the blur of a captured image with high accuracy. [Brief description of the drawings]

[0010] [Figure 1] FIG. 2 is a block diagram of an information processing system in each embodiment. [Diagram 2] 4 is a flowchart showing a process of an information processing device in each embodiment. [Diagram 3] 4 is a flowchart showing a photographing process in each embodiment. [Figure 4] 4A to 4C are explanatory diagrams of photographed images and composite images in each embodiment. [Figure 5(a)] FIG. 11 is an explanatory diagram of a photographed image list. [Figure 5(b)] FIG. 11 is an explanatory diagram of a photographed image list. [Figure 5(c)] FIG. 11 is an explanatory diagram of a photographed image list. [Figure 6] 10 is a flowchart showing a shooting condition determination process in each embodiment. [Figure 7] 5 is a flowchart showing an image quality determination process in the first embodiment. [Figure 8] FIG. 4 is an explanatory diagram of a defocus map in each embodiment. [Figure 9] FIG. 4 is an explanatory diagram of an abnormal brightness map in the first embodiment. [Figure 10] FIG. 4 is an explanatory diagram of an exclusion map in the first embodiment. [Figure 11] FIG. 4 is an explanatory diagram of a histogram of a first frequency characteristic in the first embodiment. [Figure 12] FIG. 4 is an explanatory diagram of a predetermined region in the first embodiment. [Figure 13] 11A and 11B are explanatory diagrams of images to be re-photographed in each embodiment. [Figure 14] 13 is a flowchart showing an image quality determination process in the second embodiment. [Figure 15] FIG. 11 is an explanatory diagram of a defocus map in the second embodiment. [Figure 16] FIG. 4 is an explanatory diagram of a setting screen in each embodiment. [Figure 17] FIG. 11 is an explanatory diagram of a defocus map in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same components are designated by the same reference numerals, and duplicated descriptions will be omitted. EXAMPLES

[0012] First, an information processing system 10 in this embodiment will be described with reference to FIG. 1. FIG. 1 is a block diagram of the information processing system 10. The information processing system 10 is configured to include an information processing device 100, a photography auxiliary device 150, and an imaging device 180. The information processing system 10 executes an information processing method for determining the quality of a photographed image and determining the necessity of re-photographing the image. The information processing system 10 is also a system for inspecting a structure based on an image of the structure. Examples of structures include, but are not limited to, bridges, tunnels, roads, buildings, dams, embankments, and electrical equipment.

[0013] The information processing device 100 is a device for controlling the entire photographing process of this embodiment. The information processing device 100 includes a CPU 101, a ROM 102, a RAM 103, a HDD 104, a display unit 105, an operation unit 106, and a communication unit 107.

[0014] The CPU 101 is a central processing unit that performs calculations and logical determinations for various processes and controls each component connected to a system bus 110. The CPU 101 includes a conversion unit 101a, a generation unit 101b, and a determination unit 101c.

[0015] The conversion means 101a converts the captured image into first frequency characteristic data. The generation means 101b generates second frequency characteristic data by excluding data in a predetermined region from the first frequency characteristic data. The determination means 101c determines whether the captured image is blurred from the second frequency characteristic data. The generation means 101b also determines the predetermined region depending on whether the captured image belongs to a predetermined range based on a statistic obtained based on the first frequency characteristic data.

[0016] ROM (Read-Only Memory) 102 is a program memory that stores programs for control by CPU 101, including various processing procedures described below. RAM (Random Access Memory) 103 is used as a temporary storage area such as the main memory and work area of ​​CPU 101. Note that the program memory may be realized by loading a program into RAM 103 from an external storage device connected to information processing device 100.

[0017] The HDD 104 is a hard disk for storing electronic data such as image data and programs according to this embodiment. An external storage device may be used to fulfill a similar role. Here, the external storage device may be realized, for example, by a medium (recording medium) and an external storage drive for realizing access to the medium. Known examples of such media include flexible disks (FDs), CD-ROMs, DVDs, USB memories, MOs, and flash memories. The external storage device may also be a server device connected via a network.

[0018] The display unit 105 is, for example, a liquid crystal display (LCD) or an organic light emitting display (OLED), and is a device that outputs an image on a display screen. The display unit 105 may be an external device connected to the information processing device 100 by wire or wirelessly. The operation unit 106 has a keyboard and a mouse, and accepts various operations by a user.

[0019] The communication unit 107 performs two-way wired or wireless communication with other information processing devices, communication devices, external storage devices, etc., using known communication techniques. The communication unit 107 may be configured, for example, with a chip or antenna for public wireless communication. Note that the communication unit 107 may also be configured to perform communication using other wireless communication methods, such as wireless LAN and Bluetooth (registered trademark).

[0020] In this embodiment, the photographing assistant device 150 is a camera platform device that can change the photographing position and photographing direction based on control from the information processing device 100, and is equipped with an imaging device 180 (described later). The photographing assistant device 150 has a communication unit 151, a position control unit 152 that controls the photographing position and photographing direction, and a photographing instruction unit 153.

[0021] The communication unit 151 communicates with the information processing device 100 wirelessly or via a wire, and controls the shooting direction and position and issues shooting instructions according to instructions from the information processing device. The communication unit 151 may be configured, for example, with a chip or antenna for public wireless communication. Note that the communication unit 151 may also be configured to communicate using other wireless communication methods such as wireless LAN and Bluetooth (registered trademark).

[0022] The position control unit 152 changes the photographing position and direction of the camera platform device so that the photographing area of ​​the subject to be inspected can be photographed. The photographing instruction unit 153 controls the photographing of the imaging device 180 set to the photographing position and direction changed by the position control unit 152.

[0023] The imaging device 180 is a device that captures images based on shooting instruction information received from the information processing device 100 via the shooting auxiliary device 150. The imaging device 180 preferably has a full-field phase-difference imaging sensor and also records focus level information (defocus value) of the captured image. The focus level information may be acquired by a method using infrared rays or a method using parallax images.

[0024] The focus level information will be described in detail later with reference to Figs. 8(a) and (b). The defocus value is data representing a spatial (two-dimensional) distribution of defocus amount in the shooting range. In the following description, the data representing the spatial distribution of defocus amount is also referred to as a defocus map. The defocus amount is the amount of focus deviation from the distance at which the optical system of the image capture device 180 is in focus. The defocus value (defocus map) indicates the defocus amount as a spatial distribution for each pixel of the image.

[0025] Each pixel of the full-field phase-difference imaging sensor of the imaging device 180 has two photoelectric conversion units, which are referred to as divided pixel A and divided pixel B. In the full-field phase-difference imaging sensor, images A and B are output as parallax images from divided pixels A and B that are regularly arranged two-dimensionally. An image A+B obtained by adding images A and B is recorded as a recorded still image. A defocus amount is calculated based on the phase difference between these parallax images. Note that, although a configuration in which the defocus amount is derived on a pixel-by-pixel basis will be described as an example, the defocus amount may be derived for each predetermined region, such as a block unit having a plurality of pixels (e.g., 5 pixels×5 pixels).

[0026] Next, the processing of the information processing device 100 in this embodiment will be described with reference to Fig. 2. Fig. 2 is a flowchart showing the processing of the information processing device 100. The flowchart shown in Fig. 2 is started when the information processing device 100 executes an image capture processing control application.

[0027] First, in step S201, the CPU 101 of the information processing device 100 performs a photographing process. The photographing process shown in step S201 is a process for operating the photographing assistant device 150 and the imaging device 180 and performing photographing under the control of the information processing device 100. The photographing process shown in step S201 will be described later with reference to the flowchart of FIG.

[0028] In step S201, the photographing range and photographing position are specified in the order shown in Fig. 4(a), the photographing auxiliary device 150 is controlled, and the imaging device 180 is caused to perform photographing. In step S201, when the information processing device 100 receives a specification of an inspection range (photographing range) in a structure from a user, the information processing device 100 creates a photographed image list including information indicating the photographing range or photographing position / direction of an image corresponding to each record. The photographed image list corresponds to, for example, a table 501 shown in Fig. 5(a).

[0029] Also, in step S201, the information processing device 100 causes photographing to be performed in order based on the table 501. The information processing device 100 can change the photographing range of the imaging device 180 by driving the photographing auxiliary device 150 based on the table 501, which is a list of photographed images. When the information processing device 100 drives the photographing auxiliary device 150 to change the photographing range, it transmits a photographing instruction to the imaging device 180 via the photographing auxiliary device 150 so that photographing is performed. The imaging device 180 performs photographing in response to receiving the photographing instruction.

[0030] When the information processing device 100 finishes shooting one image, it receives a shooting completion notification from the photographing assistant device 150 or the imaging device 180. When the information processing device 100 receives the shooting completion notification, it writes information such as the captured image file name to a record in the captured image list corresponding to that shooting. Then, in order to take the next image, it transmits information for changing the shooting direction by the photographing assistant device 150 and a shooting instruction to the photographing assistant device 150 and the imaging device 180, and takes the image corresponding to the next record in the list.

[0031] The image capturing process is repeated until all images corresponding to the captured image list created in this manner are captured. When all images corresponding to the captured image list have been captured, the image capturing process ends.

[0032] After the shooting process is completed, the table 501 shown in FIG. 5B is updated to include the captured image file name. The information processing device 100 is configured to transmit information for driving the shooting assistant device 150 and shooting instructions for each image, but is not limited to this. For example, the information processing device 100 may be configured to transmit information for shooting all images corresponding to records included in the table 501, which is the captured image list, to the shooting assistant device 150 and the imaging device 180 in a lump. The information processing device 100 may be configured to transmit information for shooting a plurality of images corresponding to records included in the table 501, which is the captured image list, to the shooting assistant device 150 and the imaging device 180. The shooting assistant device 150 and the imaging device 180 may be configured to transmit information to be input to the table 501, which is the captured image list, in a lump after all shooting is completed, to the information processing device 100.

[0033] Fig. 4(a) shows an imaging area (photographed image) 400 of the inspection target surface of a structure to be inspected. It indicates that the corresponding part of the inspection target surface is photographed in the direction indicated by the arrow 411 in the imaging area 400, with the imaging position 421 indicated by the diamond-shaped rectangle as the center. Fig. 4(b) shows a composite image 450 composed of the images photographed in order in Fig. 4(a). The composite image 450 is used to inspect the imaging area 400, which is the inspection target surface.

[0034] 2, the CPU 101 acquires the table 501 and the captured image file created in step S201. The subsequent processes from step S203 to step S205 are repeated for each acquired captured image file in the order of the captured image IDs listed in the table 501.

[0035] Next, in step S203, the CPU 101 performs a photographing condition determination process for determining the photographing conditions when the photographed image was photographed. The photographing condition determination process will be described later with reference to FIG. 6. In the photographing condition determination process, if the photographing conditions of the photographed image determined do not match the predetermined photographing conditions, the photographing condition is recorded as "out of photographing condition", which is information indicating that the predetermined photographing conditions are not met, in the item of determination information of the corresponding photographed image ID in the table 501. The result of this photographing condition determination process is used in a re-photographing determination process to be described later. That is, according to this embodiment, it is possible to determine whether or not re-photographing is necessary according to the photographing conditions of the photographed image. In addition, by performing the determination process of step S203 using the photographing conditions, it is possible to determine the photographed image that needs to be re-photographed without performing the image quality determination process of step S205 to be described later. In addition, by performing the photographing condition determination process of step S203, which has a lower load than the image quality determination process of step S205 to be described later, in an earlier stage, it is possible to shorten the processing time.

[0036] As an example of the photographing condition determined in step S203, the aperture value (F value) can be used. The aperture value is an important photographing condition for obtaining an image with a deep depth of field and with less influence of "blur" caused by diffraction. Therefore, in this embodiment, it is determined that an image obtained by photographing with an aperture value outside a predetermined range or value should not be used for predetermined image processing such as abnormality detection for inspection or composition processing, and it is necessary to re-photograph the corresponding photographing area.

[0037] Also, an ISO value (sensitivity) can be used as an example of the photographing condition determined in step S203. If the ISO value in the photographing condition is too high, noise is likely to be high. If an inspection is performed using an image with a lot of noise, there is a risk of accuracy decreasing. Therefore, in this embodiment, it is determined that an image obtained by photographing with an ISO value outside a predetermined range or value should not be used for predetermined image processing such as abnormality detection and composition processing for inspection, and that re-photographing of the corresponding photographing area is necessary.

[0038] Also, the subject distance can be used as an example of the photographing condition determined in step S203. If the distance to the subject during photographing is too far, there is a possibility that an abnormality in a small subject cannot be detected. Therefore, in this embodiment, it is determined that an image obtained by photographing at a subject distance other than a predetermined range or value should not be used for predetermined image processing such as abnormality detection for inspection or synthesis processing, and that it is necessary to re-photograph the corresponding photographing area. Note that, as for the photographing condition determined in step S203, it is sufficient that a condition is set for at least one condition, and conditions may be set for each of multiple conditions.

[0039] Next, in step S204, CPU 101 uses the determination result of step S203 to determine whether or not to proceed to processing of step S205 for the image to be processed. If it is determined in step S203 that the captured image meets the predetermined shooting conditions (Yes), the process proceeds to step S205. On the other hand, if the captured image does not meet the predetermined shooting conditions (No), the process proceeds to step S203 and processes the next captured image ID.

[0040] In step S205, the CPU 101 performs image quality determination processing to determine the image quality of the captured image. The image quality determination processing will be described later with reference to FIG. 7. The information processing device 100 records the determination result of the image quality determination processing in step S205 in the determination information item of the table 501 shown in FIG. 5(c). For example, if the image quality determination condition is satisfied, "OK" is recorded. On the other hand, if the image quality determination condition is not satisfied, "NG" is recorded. In this embodiment, the image quality determination processing in step S205 uses the frequency characteristics of the captured image to determine the image quality. By using the frequency characteristics, it is possible to determine the deterioration of image quality due to blurring caused by the movement of the imaging device 180 due to the influence of wind during the shooting process.

[0041] Here, "blur" includes both out-of-focus and blur. Out-of-focus refers to a state in which the subject (the wall of a structure in the case of a bridge) is out of focus. On the other hand, blur refers to a blur in a captured image caused by the movement of the imaging device itself during exposure when shooting. For example, blur can occur when the imaging device is moved vertically during exposure while held by hand, or when the imaging device is mounted on an unmanned aircraft such as a drone, due to shaking during flight.

[0042] Next, in step S206, CPU 101 determines whether or not the image quality determination process in step S205 has been completed for all captured images input in step S202. If the process has been completed for all images (Yes), the process proceeds to step S207. On the other hand, if the process has not been completed (No), the process proceeds to step S203, where the captured image with the next captured image ID is processed.

[0043] In step S207, CPU 101 determines whether or not the image quality judgment result "NG" is included in the judgment information in table 501. If the judgment information includes the image quality judgment result "NG" (Yes), the process proceeds to step S208. On the other hand, if the image quality judgment result "NG" is not included (No), the process ends.

[0044] In step S208, the CPU 101 identifies the image to be re-captured. In this embodiment, the information processing device 100 creates a composite image based on the photographing positions and photographing ranges of the photographed images, and presents the photographing positions that need to be re-captured in the composite image to the user. An image whose judgment information in the table 501 is not OK is identified, and the photographing positions and photographing ranges that need to be re-captured in the composite image are presented based on the photographing position information.

[0045] As shown in FIG. 13, the re-photographed image 1301 and the re-photographed image 1302 in the composite image 450 are displayed on the display unit 105. In the example of FIG. 13, the photographing position or photographing range that needs to be re-photographed is highlighted on the composite image to present it to the user, but other forms are also possible. For example, the photographing position and direction that needs to be re-photographed may be output as text information. Also, information that can identify the photographed image that needs to be re-photographed may be stored in a specified folder in the device. Also, information for identifying the presence of an image that should not be used for image processing such as abnormality detection and composition or the need for re-photographing may be output as a display or sound.

[0046] Also, information indicating the judgment result in step S205 may be associated with each of the judged images and output. For example, information indicating that the judged images can / cannot be used for image processing such as abnormality detection or synthesis, or that re-imaging is / is not required, may be associated with each of the judged images and output. Also, information indicating that the judged images satisfy the conditions, such as "OK", or information indicating that the image quality does not satisfy the conditions, may be associated with each of the judged images and output. In this way, information indicating whether or not an image should be used for a predetermined image processing may be output in various ways.

[0047] Next, step S201 (photographing process) in Fig. 2 will be described in detail with reference to Fig. 3. Fig. 3 is a flowchart showing the photographing process.

[0048] First, in step S301, the information processing device 100 accepts a designation of an inspection range (photographing range) of a structure from a user. The information processing device 100 determines the number of captured images and their respective photographing positions based on the area of ​​the designated inspection range (photographing range). For example, the information processing device 100 accepts a designation from a user that the photographing area 400 in FIG. 4(a) is the inspection range (photographing range). The information processing device 100 calculates the number of images to be photographed and their respective photographing positions based on two diagonal points of the photographing area 400. For example, if the inspection target of a structure is 7 m wide x 3 m long and the range that can be photographed with one photographed image is 1 m wide x 1 m long, it is determined that 21 photographed images of 7 columns x 3 rows are required, as shown in FIG. 4(b).

[0049] When the user has completed the designation of the inspection range (photographing range), in step S302, the CPU 101 creates a photographed image list shown in table 501 in FIG. 5(a) based on the number of photographed images and the photographing positions calculated in step S301. In FIG. 5(a), table 501 is a table representing the photographed image list. Table 501 is composed of photographed image IDs, photographing positions, photographed image file names, and judgment information. Note that the attributes included in table 501 are merely examples, and it is not necessary that all of these are essential, and the configuration may include other attributes. The photographed image ID is an ID for identifying the photographed image file. The photographing position represents the photographing position used in the camera platform device, and for example, the upper left is stored in the form of coordinates such as "row 1, column 1". The judgment information stores the result of judgment in the image quality judgment process in step S205. When table 501 is created in step S302, as shown in FIG. 5(a), the shooting ID calculated from the number of shots and the shooting location information are entered, and the fields for the captured image file name and judgment information are left blank.

[0050] Next, in step S303, the CPU 101 controls the camera platform device and the imaging device 180, which are the photography assistant device 150, to take pictures in the order of the captured image IDs in the table 501 created in step S302, based on the shooting position information corresponding to the captured image IDs.

[0051] The information processing device 100 changes the shooting direction and shooting position of the photographing assistant device 150 for each photographed image ID based on the coordinate information described in the photographing position information of the table 501. Then, the information processing device 100 controls the imaging device 180 to adjust the focus by an autofocus function that uses the center of the screen as a distance measuring point, for example. The information processing device 100 transmits a shooting instruction to the imaging device 180 so that shooting is performed as the autofocus is completed. In response to the control of the photographing assistant device 150 or the shooting completion notification transmitted from the imaging device 180, the information processing device 100 writes information such as the photographed image file name in the record of the photographed image list in the photographed image file name column of the table 501 as shown in FIG. 5(b). The information processing device 100 repeats such processing until the shooting of the images corresponding to the photographed image list is completed. When the shooting of all the images corresponding to the created photographed image list is completed, the shooting processing is terminated.

[0052] Next, step S203 (photographing condition determination process) in Fig. 2 will be described in detail with reference to Fig. 6. Fig. 6 is a flowchart showing the photographing condition determination process.

[0053] First, in step S601, the CPU 101 acquires shooting information (values ​​related to shooting information) included in the captured image to be processed. The shooting information is metadata recorded as shooting parameters used when the image was captured, and includes, for example, the shooting time, the aperture value (F value), the ISO sensitivity, etc., but is not limited to these.

[0054] Next, in step S602, CPU 101 determines whether or not the value related to the shooting information acquired in step S601 matches a predetermined shooting condition. If it is determined that the value related to the shooting information is outside the shooting condition (Yes), the process proceeds to step S603. On the other hand, if it is determined that the value related to the shooting information is within the shooting condition (No), the process proceeds to end processing. The shooting condition is a condition regarding whether or not the value related to the shooting information is within a threshold range, with a threshold or range set in advance. For example, the shooting condition may be the aperture value, ISO, or distance to the subject described in the processing description of step S203.

[0055] In step S603, if the determination result in step S602 is outside the shooting conditions, CPU 101 determines the determination result as outside the shooting conditions and records it in the determination information item of table 501. Note that, if the value of the acquired shooting information satisfies a predetermined shooting condition, it may record that fact in the determination information item of table 501.

[0056] Next, step S205 (image quality determination process) in Fig. 2 will be described in detail with reference to Fig. 7. Fig. 7 is a flowchart showing the image quality determination process in this embodiment.

[0057] First, in step S701, the CPU 101 (conversion means 101a) performs frequency analysis processing on the captured image to obtain frequency components (first frequency characteristics, first frequency characteristic data). That is, the conversion means 101a converts the captured image into first frequency characteristic data. In this embodiment, a wavelet transform is used as an example of frequency analysis processing to calculate horizontal and vertical frequency components. By obtaining the horizontal and vertical frequency components, it becomes possible to determine both horizontal and vertical blurring.

[0058] It is not essential to obtain both horizontal and vertical frequency components, and only horizontal or vertical frequency components may be obtained. Alternatively, a diagonal frequency component may be obtained to perform blur determination. A first feature amount obtained by performing standardization processing, threshold processing, or the like on first frequency characteristic data obtained by performing a wavelet transform or Fourier transform on a captured image may be used. The content of the processing is not limited to standardization processing or threshold processing.

[0059] In this embodiment, the frequency analysis process is not limited to the process of acquiring frequency components using wavelet transform, and frequency components may be acquired using Fourier transform. In this embodiment, the frequency characteristics of the entire captured image are acquired, but the invention is not limited to this, and frequency characteristics of only a part of the captured image may be acquired. For example, frequency characteristics of the central 50% area of ​​the captured image may be acquired. This can reduce the processing time of the wavelet transform.

[0060] Next, in step S702, CPU 101 acquires focus level information included in the image to be processed (captured image). The focus level information in this embodiment is information acquired together with an image captured by imaging device 180 having a full-field phase-difference imaging sensor, and is information in which the defocus amount for each region of the image is recorded.

[0061] 8(a) and (b) are examples of defocus maps that are focus degree information. An outer frame 801 and an outer frame 821 are frames corresponding to the image size of an input image. Numerical values ​​802 and 822 are defocus amounts indicating the focus degree in each region. A line 803 and a line 823 indicate a boundary between regions with different defocus amounts. Here, the defocus value is a numerical value of the amount of shift in focus of an object for each pixel of an image. Note that the sign of the defocus amount when the focus is shifted forward and backward in the depth direction may be changed. That is, for example, the defocus amount when the focus is shifted forward may be positive, and the defocus amount when the focus is shifted backward may be negative.

[0062] When capturing an image, if an image plane phase difference imaging sensor is arranged on the entire surface of the sensor (imaging element) of the imaging device 180, it is possible to obtain information on the defocus amount at the position of the pixel that can be obtained. Note that a known technique can be used to obtain the defocus value information. The defocus value may be obtained using a parallax image captured by a stereo camera or the like.

[0063] In the defocus maps of Figs. 8(a) and (b), the area with the defocus value "0" is an area that is in focus (in-focus area) in the focusing unit of the imaging device 180. Also, an area with a defocus amount value greater than "0" indicates that the amount of deviation from the distance at which the optical system is in focus increases (the degree of focus decreases) with each increase in the value. In the example of Fig. 8(a), the defocus amount is expressed in two stages, "0" and "1", and in the example of Fig. 8(b), it is expressed in three stages, "0", "1", and "2", but the number of stages is not limited to this and may be other values.

[0064] Next, in step S703, the CPU 101 acquires abnormal luminance information of the captured image. The abnormal luminance information is information indicating a luminance saturation region or a black crush region of the captured image. The luminance saturation region and the black crush region are low contrast regions. Such regions have many low frequency components even if no blurring occurs during shooting. Therefore, it may be determined that blurring occurs even though no blurring occurs. Therefore, in step S703, the abnormal luminance region is specified and excluded from the first frequency characteristic data. For example, when the captured image is a developed image with a luminance value of 0 to 255, the average luminance value of the entire captured image is set as a reference value, and the abnormal luminance value is set to a range of -20 to +20 from the reference value, so that the abnormal luminance region can be specified. Note that a method of setting the range from the reference value will be described with reference to a setting screen 1601 in FIG. 16 described later.

[0065] When a brightness saturation area (brightness value 255) exists, the average brightness value of the inspection target area cannot be obtained correctly. For this reason, it is preferable to obtain (calculate) the average brightness value from an area obtained by excluding the brightness saturation area from the captured image. Note that it is not essential to use the average brightness value as the reference value, and the reference value may be set in advance. FIG. 9 is an example of an abnormal brightness map, which is abnormal brightness information, in the captured image having the defocus map of FIG. 8(a). In the abnormal brightness map of FIG. 9, the area marked with "0" is an area (abnormal brightness area) that is farther away from the average brightness value of the captured image by more than the reference value. Note that the order of steps S702 and S703 may be reversed, and the abnormal brightness map may be obtained before the defocus map.

[0066] 7, the CPU 101 (the generating means 101b) removes a predetermined region from the first frequency characteristic (first frequency characteristic data) to obtain a second frequency characteristic (second frequency characteristic data). That is, the generating means 101b generates the second frequency characteristic data by removing data of the predetermined region from the first frequency characteristic data.

[0067] For this reason, first, the CPU 101 generates an exclusion map having "0" in the defocus area and the abnormal brightness area and "1" in the other areas based on the defocus map and the abnormal brightness map, and multiplies the first frequency characteristic by this exclusion map. FIG. 10 is an example of an exclusion map generated based on the defocus map of FIG. 8(a) and the abnormal brightness map of FIG. 9. It is not essential to generate an exclusion map, and it is sufficient to be able to exclude the defocus area and the abnormal brightness area, so the processing method is not limited to this. By excluding the defocus area and the abnormal brightness area in advance from the first frequency characteristic, it is possible to accurately determine a reference value (statistics) described later.

[0068] When the majority of the captured image is a defocused region and an abnormal brightness region, a peak appears on the lower frequency side than the peak frequency of the inspection target (for example, a concrete wall surface). If this peak is used as a reference value to exclude the area outside the predetermined range, the concrete wall surface of the inspection target will be excluded. In other words, the region of the captured image for which the reference value is calculated is determined based on whether or not the feature amount (second feature amount) based on the distance information belongs to a predetermined range (predetermined second range) based on the distance information on the focus plane of the captured image. In this embodiment, the region marked with "0" in the defocus map of FIG. 8(a) corresponds to the predetermined second range. Note that when the defocus map has a standardized value such as 0 to 1, for example, a region having a value from 0 to 0.2 may be set as the predetermined second range.

[0069] Moreover, the area of ​​the captured image for which the reference value is calculated is determined based on whether or not the feature based on the luminance value (third feature) belongs to a predetermined range (predetermined third range) based on the average luminance value of the captured image. In this embodiment, the predetermined range refers to a range of -20 to +20 from the reference value, with the average luminance value of the entire captured image being the reference value. In the abnormal luminance map of Fig. 9, the area corresponds to the area marked with "1".

[0070] Next, CPU 101 divides the first frequency characteristic excluding the defocused region and the abnormal luminance region into regions of 100 vertical x 150 horizontal, and calculates the average value (average wavelet value) for each region. The number of divisions of the regions is not limited to this, and the average value can be calculated from any number of divisions. Also, instead of the average value, the mode or median value may be used. Next, outlier regions are obtained based on the average value for each region. The outlier regions are regions outside a predetermined range from the reference value in the first frequency characteristic excluding the defocused region and the abnormal luminance region.

[0071] FIG. 11 is an example of a histogram display of the average wavelet value for each region. In this embodiment, the mode 1101 of the average wavelet value for each region is set as the reference value, and the predetermined range is set from −2.0 to +2.0 from the reference value. In this way, the reference value is a statistic based on the first frequency characteristic data of the captured image, that is, a statistic of the first feature amount related to the first frequency characteristic data (at least one of the average value, the median value, and the mode value). A method of setting the predetermined range from the reference value will be described with reference to a setting screen 1601 in FIG. 16 described later. In this embodiment, the statistic is based on one captured image, but the statistic may be based on multiple captured images captured in step S201. In addition, it is not essential to use the mode as the reference value, and other representative values ​​such as the average value or the median value may be used. In addition, the numerical value from the reference value that defines the predetermined range is not limited to −2.0 to +2.0.

[0072] By statistically determining the reference value based on the captured image, it becomes possible to exclude foreign objects in the inspection object having various frequency distributions, not limited to the concrete wall surface. When the inspection object is a concrete wall surface, exposed reinforcing bars, pipes, and dirt may be present in the captured image. Compared to the concrete wall surface, exposed reinforcing bars have many high-frequency components, while pipes and dirt have many low-frequency components. Therefore, if a reference value determined regardless of the inspection object is used, it will be a factor that reduces the accuracy of blur judgment. In the histogram of FIG. 11, area 1102 corresponds to the pipes and dirt, and area 1103 corresponds to the exposed reinforcing bars. The accuracy of blur judgment can be improved by making a judgment in an area excluding the areas of pipes, dirt, and exposed reinforcing bars.

[0073] 12 is an exclusion map showing the regions that are ultimately excluded from the first frequency characteristic. Regions marked with "0" are to be excluded, 1201-1203 are defocused regions, 1204 is an abnormal brightness region, and 1205-1206 are outlier regions in the frequency characteristic (1205: piping, 1206: exposed rebar). Based on this exclusion map, CPU 101 generates the second frequency characteristic from the first frequency characteristic.

[0074] In this embodiment, the predetermined region to be excluded from the first frequency characteristic preferably includes at least one of a defocus region, an abnormal brightness region, and an outlier region. The predetermined region is determined depending on whether or not the first feature amount related to the first frequency characteristic data belongs to a predetermined range (predetermined first range) based on the statistics of the first feature amount. For example, the predetermined region is a region where the first feature amount is outside the predetermined first range (a region that does not belong to a predetermined range based on the statistics obtained based on the first frequency characteristic data). If the frequency characteristic is acquired after multiplying the excluded map by the captured image, the boundary with the excluded region becomes an edge, so that high-frequency components that do not actually exist are generated. Therefore, it is preferable to multiply the excluded map by the first frequency characteristic.

[0075] 7, CPU 101 analyzes the frequency components of the second frequency characteristic and calculates an average value. Note that it is not essential to calculate an average value, and a mode or a median value may be calculated.

[0076] Next, in step S706, CPU 101 determines whether the average value (frequency components) calculated in step S705 is equal to or greater than a predetermined threshold. If the frequency components are equal to or greater than the predetermined threshold (Yes), the process proceeds to step S707. On the other hand, if the frequency components are less than the predetermined threshold (No), the process proceeds to step S708.

[0077] In step S707, CPU 101 determines that re-imaging is not necessary (OK) as a result of the image quality judgment of the captured image, records the judgment result in the judgment information item of table 501, and ends the image quality judgment process. At this time, CPU 101 may use the captured image file to be processed for image processing such as abnormality detection and composition.

[0078] In step S708, CPU 101 determines that re-shooting is necessary (NG) as a result of the image quality judgment of the captured image, records the judgment result in the judgment information item of table 501, and ends the image quality judgment process. At this time, CPU 101 determines that the captured image file to be processed should not be used for image processing such as abnormality detection and composition.

[0079] Thus, in steps S704 to SD708, the CPU 101 (determination means 101c) determines whether or not the captured image is blurred based on the second frequency characteristic data. Here, the blurring of the captured image is blurring caused by the captured image being out of focus on the subject (out-of-focus blurring) or blurring caused by the movement of the imaging device during exposure, but is not limited to these.

[0080] Note that the image quality determination process in step S205 is an example using frequency characteristics, but the image quality may be determined using the results of other image processing. For example, the image quality may be determined based on focus level information, i.e., the degree of focus / focus, or the shooting resolution. Alternatively, information indicating blur, focus level information, and shooting resolution information may be combined. In other words, the image quality determination result using focus level information, the image quality determination result using the blur level based on frequency analysis processing, and the shooting resolution may be used to determine whether or not reshooting is required within the shooting range of the captured image.

[0081] In the present embodiment, the photographing process in step S201 is performed under the control of the information processing device 100, but the present invention is not limited to this. That is, the information processing device 100 may omit the process in step S201 and, when acquiring the photographed images in step S202, acquire information required for the subsequent processes, such as a photographed image list. EXAMPLES

[0082] Next, an information processing system according to a second embodiment of the present invention will be described. In the first embodiment, an example in which the frequency characteristics are used in the image quality determination process has been described. As described above, the image quality may be determined based on focus level information, i.e., the degree of focus / focus, or the shooting resolution. In this embodiment, the image quality determination process is performed by combining the blur determination using the frequency characteristics, the focus level information, and the shooting resolution information. Note that this embodiment is similar to the first embodiment except for the image quality determination process, and therefore a description common to the first embodiment will be omitted.

[0083] The image quality determination process in this embodiment will be described with reference to Fig. 14. Fig. 14 is a flowchart showing the image quality determination process in this embodiment.

[0084] First, in step S1401, CPU 101 acquires shooting resolution information included in the captured image. More specifically, CPU 101 calculates the shooting resolution based on the image size of the captured image, the size of the imaging sensor, and the distance to the shooting plane. The distance to the shooting plane is obtained by acquiring the distance to the subject when focusing at the position of the range-finding point of the subject.

[0085] Next, in step S1402, CPU 101 determines whether the shooting resolution acquired in step S1401 is equal to or greater than a predetermined threshold. If the shooting resolution is equal to or greater than the predetermined threshold (Yes), the process proceeds to step S1403. On the other hand, if the shooting resolution is smaller than the predetermined threshold (No), the process proceeds to step S1411. This makes it possible to determine whether the shooting resolution satisfies the required quality of the inspection image. If the shooting resolution is smaller than the threshold, that is, if the subject is roughly photographed, the result may be determined as "NG" without carrying out the subsequent determination process. Setting the threshold for determining the shooting resolution will be described below with reference to a setting screen 1601 in FIG. 16.

[0086] In step S1403, CPU 101 determines whether or not to continue the process after the shooting resolution determination. If the process is to be continued (Yes), the process proceeds to step S1404. On the other hand, if the process is to be ended, the process proceeds to step S1410. Note that the setting of whether or not to continue the process after the shooting resolution determination will be described later with reference to a setting screen 1601 in FIG.

[0087] In step S1404, the CPU 101 acquires the focus level information contained in the image to be processed. The acquisition method is the same as the processing described in the first embodiment, and therefore the description will be omitted.

[0088] Next, in step S1405, CPU 101 calculates the proportion of in-focus areas (proportion of areas where the degree of focus is less than a predetermined value) using the focus level information acquired in step S1404. The calculation of the proportion of in-focus areas will be described with reference to FIG. 8(a). The proportion of in-focus areas in an image is calculated by using the number of pixels surrounded by outer frame 801 as the denominator and the number of pixels with a defocus value of "0" as the numerator. In the example shown in FIG. 8(a), the proportion of areas where the defocus amount is "0" is approximately 90%. In the example shown in FIG. 8(b), the proportion of areas where the defocus amount is "0" is approximately 60%.

[0089] Next, in step S1406, the CPU 101 determines whether the ratio calculated in step S1405 is equal to or greater than a predetermined threshold. If the ratio is equal to or greater than the predetermined threshold (Yes), the process proceeds to step S1407. On the other hand, if the ratio is less than the predetermined threshold (No), the process proceeds to step S1411. If the ratio threshold is 80%, the ratio is equal to or greater than the threshold (80%) in FIG. 8(a), and is less than the threshold (80%) in FIG. 8(b). Note that the threshold has been described as 80%, but it may be set arbitrarily based on an input by a user. The threshold may also be determined based on the relationship between the ratio of the focused area and the detection accuracy for detecting anomalies. For example, various methods may be adopted, such as setting in advance a ratio at which the average accuracy for detecting anomalies is 95% or more as the threshold.

[0090] In step S1405, the ratio of the area where the defocus amount is "0" is calculated, but the present invention is not limited to this. For example, the ratio of the area where the defocus amount is "0" or "1" may be calculated. That is, the area where the focus degree is equal to or greater than a threshold may be calculated. In addition, in step S1405, the ratio of the area where the defocus amount is "3" may be calculated, and if the ratio is less than the threshold, the process proceeds to step S1407, and if the ratio exceeds the threshold, the process proceeds to step S1411. In addition, in step S1405, a first ratio of the area where the defocus amount is "0" and a second ratio of the area where the defocus amount is "3" may be calculated. In this case, if the first ratio exceeds the first threshold and the second ratio is less than a second threshold different from the first threshold, the process proceeds to step S1407, and otherwise the process proceeds to step S1411.

[0091] In this way, it is possible to determine whether the degree of focus of an image satisfies various predetermined conditions based on the focus degree information, and to determine whether the image should be used for image processing such as abnormality detection and synthesis according to the result of the focus state determination. It is also possible to determine whether the degree of focus of an image satisfies various predetermined conditions based on the focus degree information, and to use the result of the focus state determination to determine whether re-imaging is necessary.

[0092] In step S1407, CPU 101 determines whether or not to continue the processing after the focus level determination processing. If the processing is to be continued (Yes), the process proceeds to step S1408. On the other hand, if the processing is to be ended (No), the process proceeds to step S1410. Note that, as in step S1403, the setting of whether or not to continue the processing will be described later with reference to a setting screen 1601 in FIG. 16.

[0093] In step S1408, the CPU 101 performs frequency analysis processing on the captured image to acquire frequency components. Note that the frequency analysis processing is omitted since it is the same as in the first embodiment.

[0094] Next, in step S1409, the CPU 101 determines whether the frequency components calculated in step S1408 are equal to or greater than a predetermined threshold. If the frequency components are equal to or greater than the threshold (Yes), the process proceeds to step S1410. On the other hand, if the frequency components are less than the threshold (No), the process proceeds to step S1411. As a result, if the frequency component value is greater than the threshold, it is determined that there are many high-frequency components and many areas including edges. If the frequency component value is less than the threshold, it is determined that there are many low-frequency components and that the influence of blur is large. By performing blur determination processing after the focus degree determination, it is possible to determine whether an image is suitable as an inspection image even in a shooting situation where it is determined that the image is in focus but blur occurs. As an example of a situation where the image is in focus but blur occurs, there is a situation where the image is taken while mounted on an aircraft such as a drone, and it is assumed that the drone shakes due to the influence of wind during shooting. The threshold for blur determination will be described using a setting screen 1601 in FIG. 16, which will be described later.

[0095] Next, in step S1412, CPU 101 generates an image on which the determination result is superimposed, using information on the result of the determination in step S1407 or step S1411. Here, the image generated in step S1512 will be described with reference to FIG. 15. FIG. 15 is an explanatory diagram of a defocus map. A resultant image 1500 has a defocus area 1501, a defocus area 1502, a defocus area 1503, and a frame area 1511. The frame area 1511 has a shooting resolution determination area 1512, a focus level determination area 1513, and a blur determination area 1514.

[0096] Defocus areas 1501, 1502, and 1503 are areas obtained by dividing the defocus map shown in FIG. 8(b) by the defocus amount, and an area with a defocus amount of 0 is represented as a defocus area 1501. Similarly, an area with a defocus amount of 1 is represented as a defocus area 1502, and an area with a defocus amount of 2 is represented as a defocus area 1503, and areas with the same defocus amount are treated as the same defocus area. A colored layer is generated for each defocus area, and the colored layer is superimposed on the captured image. For example, a warning color may be used as the color of the layer, and blue or green may be used as a color indicating high safety when the defocus amount is smaller than a predetermined value, and yellow or red may be used as a color indicating high alertness when the defocus amount is larger than a predetermined value. This makes it easier to visually recognize areas with a large defocus amount and requiring re-imaging.

[0097] In the frame area 1511, a shooting resolution determination area (shooting resolution determination icon) 1512, a focus level determination area (focus level determination icon) 1513, and a blur determination area (blur determination icon) 1514 are displayed as marks representing each determination result. The result image 1500 shows an example in which the shooting resolution determination and focus level determination are OK, but the blur determination is NG, and the expression changes depending on the result. As an example of the icon expression, the shooting resolution determination area 1512 and focus level determination area 1513 that are judged as OK are represented by black characters with white outline, and the blur determination area 1514 that is judged as NG is represented by white characters with black outline.

[0098] This not only allows the judgment result to be visually confirmed, but also makes it easier to confirm at what stage an error occurred. Also, depending on the presence or absence of an icon, it may be possible to confirm which judgment method was not executed. For example, a case will be described in which the CPU 101 judges in step S1402 in FIG. 14 that the shooting resolution acquired in step S1401 is equal to or lower than a predetermined threshold. In this case, only the shooting resolution judgment area 1512 may be displayed in the frame area 1511 of the resultant image 1500, and the focus level judgment area 1513 and the blur judgment area 1514 may not be displayed.

[0099] Also, in the frame area 1511, the color of each icon may be changed based on the judgment result. For example, red is assigned to the shooting resolution judgment, blue to the focus judgment, and yellow to the blur judgment, and if an image is NG in the focus judgment, an image is created with the frame area filled in blue. This makes it possible to distinguish judgment results that are difficult to distinguish from the icons in the judgment area by coloring the frame area, even for small image sizes such as thumbnail images. This concludes the explanation of the flowchart in FIG. 14.

[0100] 16 is a diagram showing a setting screen 1601 which is an example of a UI screen for setting parameters in this embodiment. The setting screen 1601 is made up of a judgment pattern selection section 1611 and a superimposed image storage condition selection section 1630. The judgment pattern selection section 1611 is divided into three sections: a shooting resolution judgment section 1612, a focus level judgment section 1613, and a blur judgment section 1614. Each judgment section has a judgment name label 1615, 1616, 1617, and a judgment symbol 1618, 1619, 1620, respectively.

[0101] 14, input areas 1621-1628 for values ​​such as the determination threshold used in the flowchart are arranged in 1612-1614 in correspondence with the determination processes using them. More specifically, the imaging resolution determination section 1612 is arranged with an imaging resolution determination name label 1615, an imaging resolution determination symbol 1618, and an input area 1621 for the imaging resolution determination threshold. The focus level determination section 1613 is arranged with a focus level determination name label 1616, a focus level determination symbol 1619, an input area 1622 for the focus level determination threshold, an input area 1623 for the focus level determination ratio, and an input area 1624 for the focus level determination area. The blur determination section 1614 is arranged with a blur determination name label 1617, a blur determination symbol 1620, an input area 1625 for the blur determination threshold, an input area 1626 for the blur determination area, a blur determination brightness range 1627, and a blur determination frequency range 1628. This arrangement allows the user to intuitively understand which determination process each numerical value relates to.

[0102] Also, by matching the shapes of the icons (1512, 1513, 1514) in FIG. 15 with the judgment symbols 1618, 1619, 1620, respectively, the user can intuitively grasp the relationship between the judgment settings made on this screen and the results on the result superimposed image. Of course, these icons and judgment symbols may have different designs. The setting screen 1601 is a screen displayed on a PC, smartphone, tablet terminal, etc. The user can turn on / off check boxes and radio buttons, which will be described later, and input numerical values ​​by mouse operation, touch operation, keyboard operation, etc. (by the operation unit 106).

[0103] Of the judgment name labels, focus level judgment name label 1616 consists of a focus level judgment checkbox 1616a and a focus level judgment name section 1616b. Blur judgment name label 1617 consists of a blur judgment checkbox 1617a and a blur judgment name section 1617b. By turning these checkboxes ON / OFF, it is possible to specify whether or not to further carry out the corresponding judgment process when the result of the shooting resolution judgment, which is always carried out, is OK. For example, in Fig. 16, focus level judgment checkbox 1616a is ON and blur judgment checkbox 1617a is OFF, indicating that when the result of the shooting resolution judgment is OK, focus level judgment will continue and blur judgment will not be carried out.

[0104] 16, the focus level determination check box 1616a is ON, so input fields 1622 to 1624 are available for input. In addition, the blur determination check box 1617a is OFF, so input fields 1625 and 1626 are grayed out, indicating that input is unavailable. This allows the user to focus only on inputting values ​​related to the determination to be made.

[0105] Based on the state of the focus level determination check box 1616a and the blur determination check box 1617a, the CPU 101 determines whether to continue the process in steps S1403 and S1407. As a result, even if the previous determination is determined to be OK, the subsequent process can be omitted in a situation where the subsequent process is not necessary. For example, in the case of a shooting mode using a drone in which blurring is likely to occur, the blur determination check box 1617a can be turned ON to perform blur determination. On the other hand, in the case of a shooting mode using a tripod in which blurring is unlikely to occur, the blur determination check box 1617a can be turned OFF to prevent blur determination. In addition, when only the shooting resolution determination is performed, the focus level determination check box 1616a and the blur determination check box 1617a can be turned OFF to know the shooting resolution and confirm whether an image matching the inspection image has been captured. When performing the determination process on a large number of images, the process can be terminated according to the use or purpose, and the effect of shortening the processing time as described above can be expected.

[0106] The value input to imaging resolution determination threshold input area (first setting means) 1621 represents the threshold in step S1402. The value input to focus level determination threshold input area 1622 represents a predetermined value in step S1405. The value input to focus level determination ratio input area (second setting means) 1623 represents the ratio threshold in step S1406. The value input to focus level determination region input area (second setting means) 1624 is an item for setting the region for calculating the ratio in step S1405, and represents the percentage of the area of ​​the central region to be determined when the area of ​​the entire image to be determined is taken as 100%. By making the central region to be determined smaller (narrowing the area), the amount of calculation processing is reduced, and speed improvement is expected. Furthermore, by setting the area to be judged to an arbitrary value less than 100% (for example, 50%) using the value input to the focus degree judgment area input area (second setting means) 1624, only the central part of the image to be applied to the composite image is judged when creating a stitched composite image. By only judging the central part, the outer periphery of the image that will become the overlapping portion is excluded from the judgment. This makes it possible to judge that the image can be used for stitching even if the outer periphery is out of focus.

[0107] The value input to the input area (third setting means) 1625 of the blur determination threshold represents the threshold in step S1409. The value input to the input area (third setting means) 1626 of the blur determination area is an item for setting the area for calculating frequency component values ​​in step S1408. This value represents the percentage of the area of ​​the central area to be calculated when the area of ​​the entire image to be calculated is taken as 100%. The same effect as with the focus degree determination area can be obtained.

[0108] The value input to the blur determination luminance range 1627 is used to exclude abnormal luminance regions in steps S703 and S1408. For example, if the value is 20, the region outside the range of -20 to +20 from the reference value becomes the abnormal luminance region. The value input to the blur determination frequency range 1628 is used to exclude outlier regions in steps S704 and S1408. For example, if the value is 2.0, the region outside the range of -2.0 to +2.0 from the reference value is excluded from the blur determination region.

[0109] In this embodiment, the image quality judgment process in step S205 is replaced with the flowchart shown in FIG. 14, and the shooting resolution judgment is performed before the focus judgment, and the blur judgment process is performed after the focus judgment. The shooting resolution judgment makes it possible to judge whether the image is suitable for shooting an inspection image, and the processing time can be shortened, and a large number of images can be judged in a short time. In addition, by performing the blur judgment, it is possible to judge the influence of the blur in the case of shooting using a drone, where blur occurs even if it is judged that the focus is correct. In addition, the processing time can be shortened according to the shooting environment by selecting whether or not to execute the judgment process when each judgment process is NG.

[0110] In this embodiment, as shown in Fig. 14, the processing is performed in the order of shooting resolution determination, focus level determination, and blur determination, but any order may be used. However, by performing the processing in the order of Fig. 14, the following effects can be obtained. By performing shooting resolution determination first, it is possible to determine whether to continue or end the processing depending on whether the shooting resolution of the image is suitable for inspection. Also, by performing blur determination processing last, it is possible to determine whether an image is blurred even if it is in focus. Also, by providing frequency analysis processing, which takes a lot of processing time, at the last stage, the number of times the frequency analysis processing is performed can be reduced depending on the determination results of the previous stage, thereby shortening the overall processing time.

[0111] In the screen of FIG. 16, the order of the shooting resolution determination unit 1612, the focus level determination unit 1613, and the blur determination unit 1614 enables the user to intuitively understand that these three determination processes are performed in the order listed on the setting screen 1601.

[0112] Also, the photographing resolution determination unit 1612 is not provided with checkboxes such as the focus level determination checkbox 1616a and the blur determination checkbox 1617a. This is because if the photographing resolution is not what is required for the quality of the inspection image, that is, if the subject is photographed roughly, there is no need to perform the subsequent determination process.

[0113] Furthermore, when the focus level determination checkbox (first selection means) 1616a and the blur determination checkbox (second selection means) 1617a are both OFF and the blur determination checkbox 1617a is turned ON, the focus level determination checkbox 1616a is also turned ON. When both are ON and the focus level determination checkbox 1616a is turned OFF, the blur determination checkbox 1617a may also be linked to be turned OFF.

[0114] This makes it possible to guarantee that the focus is determined before the blur determination is performed. The reason for adopting such a setting is that, as described above, by performing the blur determination process after the focus determination, even if the image is determined to be in focus, it is possible to determine whether it is suitable as an inspection image even in a shooting situation where blur is occurring.

[0115] Regarding the generation of the judgment result superimposed image, an example is shown in which it is created regardless of whether the judgment result is OK or NG, but it is also possible to generate a superimposed image only for objects judged as NG, and it is also possible to shorten the processing time without generating a superimposed image by creating a list of the judgment results. The selection section 1630 for the superimposed image saving conditions in FIG. 16 is an example of such selection using radio buttons. In FIG. 16, only "NG" is selected, but "Do not save" or "Save all" may be selected depending on the user's operation.

[0116] When "Do not save" is selected in the selection section 1630 for the superimposed image saving condition, the judgment process shown in FIG. 14 is performed, and the image judged as OK is stored in a predetermined folder, and the image judged as NG is stored in another folder. However, an image with the image quality judgment result superimposed thereon as shown in FIG. 15 is not created. When only "NG" is selected, the judgment process shown in FIG. 14 is performed, and the image judged as OK is stored in a predetermined folder, and the image judged as NG is stored in another folder. In addition, the folder in which the image judged as NG is stored also stores the image with the image quality judgment result superimposed thereon as shown in FIG. 15. When "Save all" is selected, the judgment process shown in FIG. 14 is performed, and the image judged as OK is stored in a predetermined folder, and the image judged as NG is stored in another folder. In addition, the folder in which the image judged as OK is stored also stores the image with the image quality judgment result superimposed thereon as shown in FIG. 15, and the folder in which the image judged as NG is stored also stores the image with the image quality judgment result superimposed thereon as shown in FIG. 15. By adopting such settings, it is possible to shorten the processing time while generating only the necessary images according to the user's request.

[0117] In step S1412, when generating the judgment result superimposed image, an icon indicating the judgment result is superimposed on the result image, so that it is possible to know in which process the judgment is NG. Furthermore, since no icon is displayed for processes that have not been subjected to judgment processing, it is possible to know to what extent the processing has been performed.

[0118] Next, examples of other display forms of the resultant superimposed image will be described with reference to Fig. 17(a) to (c). However, explanations of the same reference numerals as in Fig. 15 will be omitted, and only differences will be described.

[0119] 17A shows an example of a result image 1700 in which the image has been judged as NG in the blur judgment process as a result of performing the imaging resolution judgment, focus level judgment, and blur judgment. A left frame area 1711 displays an imaging resolution information area 1721, an imaging resolution judgment area 1731, a focus level judgment area 1732, and a blur judgment area 1733, while a right frame area 1712 displays an NG judgment area 1741. The imaging resolution information area 1721 displays the value of the imaging resolution information acquired in step S1401 (0.5 in FIG. 17A).

[0120] 17(a), three determinations have been made as described above, and therefore a photographing resolution determination area 1731, a focus level determination area 1732, and a blur determination area 1733 are displayed. The photographing resolution determination area 1731, the focus level determination area 1732, and the blur determination area 1733 are each displayed in a different color, and in this embodiment, the photographing resolution determination area 1731 is assigned red, the focus level determination area 1732 is assigned blue, and the blur determination area 1733 is assigned yellow.

[0121] The same color as the judgment area corresponding to the judgment process that is NG is displayed in an NG judgment area 1741 in the right frame area 1712. Since the blur judgment process is NG in FIG. 17(a), the NG judgment area 1741 is displayed in the same color as the blur judgment area 1733.

[0122] A result image 1750 in Fig. 17(b) is an example in which the result of performing the photographing resolution determination and the focus level determination is NG in the focus level determination process, and no blur determination is performed. In Fig. 17(b), a photographing resolution determination area 1731 and a focus level determination area 1732 are displayed in the left frame area 1711, and a blur determination area 1733 is not displayed. This is because, as described above, the result of performing the photographing resolution determination and the focus level determination is NG in the focus level determination process, and no blur determination is performed. An NG determination area 1741 in the right frame area 1712 is displayed in the same color as the focus level determination area 1732 corresponding to the focus level determination that was NG.

[0123] A result image 1770 in Fig. 17(c) is an example of a case where none of the photographing resolution judgment, focus judgment, and blur judgment results in a NG judgment. In Fig. 17(c), a photographing resolution judgment area 1731, a focus judgment area 1732, and a blur judgment area 1733 are displayed in a left frame area 1711, and a NG judgment area 1741 is not displayed in a right frame area 1712.

[0124] If the shooting resolution judgment is NG, only the shooting resolution judgment area 1731 is displayed in the left frame area 1711, and the NG judgment area 1741 in the right frame area 1712 is displayed in the same color as the shooting resolution judgment area 1731. Also, if only the focus degree judgment check box 1616a is turned ON in the judgment pattern selection unit 1611 and neither the shooting resolution judgment nor the focus degree judgment is NG, the shooting resolution judgment area 1731 and the focus degree judgment area 1732 are displayed in the left frame area 1711. On the other hand, nothing is displayed in the NG judgment area 1741 in the right frame area 1712.

[0125] With this type of UI, it is possible to check the shooting resolution information in the result image. In addition, it is possible to check which of the three judgment processes, shooting resolution judgment, focus judgment, and blur judgment, was actually performed, and intuitively check which process was NG. In other words, it is possible to generate visualization information that visualizes which of the shooting resolution judgment, focus judgment, and blur judgment was performed, and which of the performed judgments was determined not to satisfy a predetermined condition.

[0126] The colors of the imaging resolution determination area 1731, the focus level determination area 1732, and the blur determination area 1733 may correspond to the colors of the imaging resolution determination symbol 1618, the focus level determination symbol 1619, and the blur determination symbol 1620 in Fig. 16. This makes it possible for the user to intuitively grasp the relationship between the determination set by the user and the result on the result superimposed image.

[0127] In this embodiment, an example of an information processing device that includes a determination means having the functions of focus determination, blur determination, and shooting resolution determination and can select a determination mode has been described. However, the determination means may be configured to have only one or two of these functions. For example, the determination means may be configured to perform only the shooting resolution determination among the above functions, or may be configured to perform only the focus determination and shooting resolution determination. The configuration of the acquisition means may be matched to the configuration of the determination means. Even with such a configuration, the information processing device can determine whether the captured image should be used for predetermined image processing based on the image quality of the captured image.

[0128] <Other Examples> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-mentioned embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0129] According to each embodiment, it is possible to provide an information processing method, an information processing device, and a program capable of determining the blur of a captured image with high accuracy.

[0130] The disclosure of each embodiment includes the following configurations and methods. (Method 1) A conversion step of converting the captured image into first frequency characteristic data; a generating step of generating second frequency characteristic data by excluding data in a predetermined region from the first frequency characteristic data; and determining whether or not the captured image is blurred based on the second frequency characteristic data. An information processing method characterized in that, in the generating step, the specified area is determined depending on whether or not it belongs to a specified range based on a statistical quantity obtained based on the first frequency characteristic data. (Method 2) The information processing method described in Method 1, characterized in that the blur of the captured image determined in the determination step is blur caused by the captured image not being in focus on the subject, or blur caused by movement of the imaging device during exposure. (Method 3) A first feature amount is obtained by performing a standardization process or a threshold process on the first frequency characteristic data, The information processing method according to method 1 or 2, wherein the statistics are obtained based on the first feature. (Method 4) An information processing method described in any one of methods 1 to 3, characterized in that the area of ​​the captured image for calculating the statistical quantity is determined based on whether a second feature based on distance information of the captured image belongs to a predetermined range based on the distance information in the focus plane of the captured image. (Method 5) An information processing method described in any one of methods 1 to 3, characterized in that the area of ​​the captured image for calculating the statistical quantity is determined based on whether a third feature based on a luminance value of the captured image belongs to a predetermined range based on an average luminance value of the captured image. (Method 6) The information processing method according to method 5, characterized in that the average brightness value is obtained from an area excluding brightness saturated areas from the captured image. (Method 7) The information processing method according to any one of Methods 1 to 6, wherein the statistical quantity is at least one of a mean, a median, and a mode. (Method 8) The information processing method according to any one of Methods 1 to 7, wherein the first frequency characteristic data is generated by performing a wavelet transform or a Fourier transform on the captured image. (Method 9) The information processing method described in Method 3, characterized in that the specified area is an area in which the first feature does not belong to a specified range based on a statistical quantity obtained based on the first frequency characteristic data. (Configuration 1) A conversion means for converting a captured image into first frequency characteristic data; a generating means for generating second frequency characteristic data by excluding data in a predetermined region from the first frequency characteristic data; a determination means for determining whether or not the captured image is blurred based on the second frequency characteristic data, The information processing device, wherein the generating means determines the predetermined region depending on whether the predetermined region belongs to a predetermined range based on a statistic obtained based on the first frequency characteristic data. (Configuration 2) 10. A program for causing a computer to execute the information processing method according to any one of Methods 1 to 9.

[0131] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0132] 100 Information processing device 101a Conversion Method 101b Generating means 101c Determination means

Claims

1. a conversion step of converting the captured image into first frequency characteristic data; a generating step of generating second frequency characteristic data by excluding data in a predetermined region from the first frequency characteristic data; and determining whether the captured image is blurred based on the second frequency characteristic data. An information processing method characterized in that, in the generating step, the specified area is determined depending on whether or not it belongs to a specified range based on a statistical quantity obtained based on the first frequency characteristic data.

2. 2. The information processing method according to claim 1, wherein the blur of the captured image determined in the determination step is blur caused by the captured image not being in focus on the subject, or blur caused by movement of an imaging device during exposure.

3. A first feature amount is obtained by performing a standardization process or a threshold process on the first frequency characteristic data, 3. The information processing method according to claim 1, wherein the statistics are obtained based on the first feature amount.

4. The information processing method described in claim 1 or 2, characterized in that the area of ​​the captured image for calculating the statistical quantity is determined based on whether a second feature based on distance information of the captured image belongs to a predetermined range based on the distance information in a focus plane of the captured image.

5. The information processing method according to claim 1 or 2, characterized in that the area of ​​the captured image for calculating the statistical amount is determined based on whether a third feature based on a luminance value of the captured image belongs to a predetermined range based on an average luminance value of the captured image.

6. The information processing method according to claim 5 , wherein the average luminance value is obtained from an area of ​​the photographed image excluding an area of ​​saturated luminance.

7. 3. The information processing method according to claim 1, wherein the statistical amount is at least one of a mean value, a median value, and a mode value.

8. 3. The information processing method according to claim 1, wherein the first frequency characteristic data is generated by performing a wavelet transform or a Fourier transform on the captured image.

9. The information processing method according to claim 3 , wherein the predetermined region is a region in which the first feature does not belong to a predetermined range based on a statistical quantity obtained based on the first frequency characteristic data.

10. A conversion means for converting a captured image into first frequency characteristic data; a generating means for generating second frequency characteristic data by excluding data in a predetermined region from the first frequency characteristic data; a determination means for determining whether the captured image is blurred or not based on the second frequency characteristic data, The information processing apparatus according to claim 1, wherein the generating means determines the predetermined region depending on whether the predetermined region belongs to a predetermined range based on a statistic obtained based on the first frequency characteristic data.

11. 3. A program for causing a computer to execute the information processing method according to claim 1.

Images