Information processing device, imaging system, method, and program

Abstract

To efficiently capture a subject that needs to be re-captured.SOLUTION: An information processing device stores image quality information for each of a plurality of images in which a subject is captured, determines whether the image quality of the images is good or not on the basis of the image quality information of the images, synthesizes a synthesized image using an image whose image quality is determined to be good, and notifies how to recapture the subject corresponding to a pixel-deficient region when a region of missing pixels on the synthesized image exists.SELECTED DRAWING: Figure 1A

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a photographing system, a method, and a program.

Background Art

[0002] Inspectors of structures such as bridges and tunnels check the position and degree of deformation (such as cracks and water leakage) on the surface of the structure and record the inspection results. Generally, a civil engineer visually checks the presence or absence of deformation on the surface of the structure and records the position and size of the deformation in a notebook or the like. However, in recent years, due to the aging of civil engineers and the like, there has been a problem of a shortage of civil engineers for inspecting structures. To address this problem, "image-based inspection" is being carried out, in which the surface of a structure is photographed in high definition with an imaging device (camera), and deformation is confirmed and recorded from the photographed image by image analysis technology.

[0003] In image-based inspection, a lens-exchangeable high-pixel and high-definition imaging device (camera) is mounted on an automatic pan-tilt unit and a drone, etc., and the surface of the structure to be inspected is comprehensively photographed using the imaging device. When photographing the surface of a structure such as a bridge and a tunnel at a resolution capable of detecting deformation (cracks), dozens to hundreds of photographed images can be obtained. However, since each photographed image shows only a narrow range of the structure, the inspector cannot grasp which range of the structure was photographed from each photographed image. Therefore, a large-scale stitched image (composite image) showing a wider range of the surface of the structure is created by stitching (compositing) each photographed image. Thereby, the photographed images can be synthesized by overlapping the respective photographed ranges at the time of photographing.

[0004] Patent Document 1 discloses a technique for determining an abnormality such as a defect in a photographed image taken by a photographing device mounted on an unmanned aerial vehicle, and specifying an image that requires re-photographing according to the presence or absence of the abnormality. Specifically, an image to be re-photographed is specified by detecting data abnormalities such as loss of image data during communication of the photographed image.

Prior Art Documents

[0005] [Patent Document 1] Patent No. 6619761 [Overview of the project] [Problems that the invention aims to solve]

[0006] However, Patent Document 1 does not provide a method for identifying a re-captured image based on the image quality of the captured image, nor a method for re-captured a re-captured image.

[0007] Thus, there is a challenge in efficiently photographing subjects that require reshoots.

[0008] The present invention aims to efficiently photograph subjects that require re-photography. [Means for solving the problem]

[0009] To achieve the object of the present invention, an information processing device according to one embodiment of the present invention has the following configuration. That is, the information processing device includes a storage means for storing image quality information for each of a plurality of images in which a subject is depicted; a determination means for determining whether or not the image quality of the images is good based on the image quality information of the images; a synthesis means for synthesizing a composite image using the images that the determination means has determined to have good image quality; and a notification means for notifying a method for re-photographing the subject corresponding to the pixel loss area if there is a pixel loss area in the composite image. [Effects of the Invention]

[0010] According to the present invention, subjects that require re-photography can be photographed efficiently. [Brief explanation of the drawing]

[0011] [Figure 1A] A diagram showing an example of the configuration of functions in an information processing device. [Figure 1B]This figure shows an example of judgment information used to determine the image quality of captured image data. [Figure 2] A diagram showing an example of a composite image created by an information processing device. [Figure 3] A diagram illustrating an example of a re-imaging method presented by an information processing device. [Figure 4] A flowchart illustrating the processing flow by an information processing device. [Modes for carrying out the invention]

[0012] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0013] (First Embodiment) The information processing device stores the image quality information of each of the multiple images of the subject and determines whether the image quality is good or not based on the image quality information. The information processing device synthesizes a composite image using the images that have been determined to have good image quality, and if there are pixel loss areas in the composite image, it notifies the user of a method for re-shooting the subject corresponding to the pixel loss area. The information processing device is mounted on a mobile body (drone) or a pan / tilt head, but it may also be mounted on any device capable of mounting an information processing device. Furthermore, the information processing device may not be mounted on the mobile body (drone) or pan / tilt head, and a separately prepared information processing device may control the mobile body (drone) or pan / tilt head by wireless communication or the like. Furthermore, data may be exchanged between the mobile body (drone) or pan / tilt head and the information processing device via a detachable storage medium. In this case, it becomes a shooting system comprising an information processing device and a mobile body (drone). Here, image quality information refers to multiple elements (e.g., shooting resolution, focus, and blur) used to evaluate the image quality of an image of a subject. Image quality refers to the quality of an image of a subject. The quality of the image is determined based on whether at least one of the multiple elements of the image quality information is good or not. A composite image is a single image created by combining multiple images of the same subject.

[0014] The following explains why technology is needed to support inspectors in judging image quality and deciding whether to retake images. For example, when inspecting extremely fine cracks with a width of 0.2 mm on the surface of a structure, if the imaging device (camera) is not set up properly, the extremely fine cracks may not be visible in the image. In particular, the shooting resolution (actual size corresponding to one pixel of the image), focus, and blur are important for ensuring image quality. When an inspector inspects cracks with a width of 0.2 mm or more, a shooting resolution of 0.5 mm / pixel or higher is sufficient. When an inspector inspects cracks with a width of 0.05 mm or more, a shooting resolution of 0.3 mm / pixel or higher is sufficient. It is possible to photograph cracks narrower than the shooting resolution, but to photograph even narrower cracks, it is necessary to photograph the cracks at a higher shooting resolution. For example, in bridge inspections under national jurisdiction, cracks wider than 0.05 mm are subject to inspection, and it is recommended to use a resolution higher than 0.3 mm / pixel for imaging (Ministry of Land, Infrastructure, Transport and Tourism (March 2021, Draft Manual for the Delivery of 3D Deliverables Using Inspection Support Technology (Image Measurement Technology))).

[0015] Focus adjustment significantly affects how cracks appear in captured images. For example, in images of cracks taken out of focus, very fine cracks will not be visible due to blurring. In image-based inspection, since the entire surface of the structure (hereinafter referred to as the inspection surface) must be captured in the image, it is necessary to set the imaging device so that the entire image is in focus. In other words, the aperture (F-number) must be set so that the depth of field (the range in focus) includes the nearest point to the furthest point of the inspection surface. Increasing the F-number increases the depth of field and widens the range in focus. However, increasing the F-number slows down the shutter speed, making the subject (cracks) more prone to blurring.

[0016] Furthermore, extremely large f-numbers can cause a type of blur called "diffraction blur," which affects how cracks are captured. Camera shake when photographing cracks also affects how the cracks are captured. In blurry images, cracks can be difficult to distinguish from dirt on the surface of the structure. Increasing the ISO sensitivity can increase the shutter speed of the camera, but increasing the ISO sensitivity too much will generate ISO noise. Since ISO sensitivity affects how cracks are captured in the image, it is necessary to adjust the shooting conditions to capture images without blur while keeping the ISO sensitivity value as low as possible.

[0017] As described above, there are clear points to consider when photographing structures during inspection, but the optimal settings cannot be determined in advance as they vary depending on the shooting conditions. For example, the focal length required to achieve the necessary shooting resolution changes depending on the distance from the imaging device (camera) to the subject (crack). The required depth of field changes depending on the orientation of the surface of the subject (crack), so the optimal F-number will differ. The brightness around the subject (crack) also changes depending on the weather, so the shutter speed required to capture a blur-free image also changes. For example, when photographing the surface of a structure with an imaging device (camera) mounted on a drone, the distance between the imaging device (camera) and the subject, as well as the orientation of the subject surface, may change just before shooting depending on the wind conditions at the time of shooting. Also, a sudden gust of wind during shooting can change the optimal shutter speed for blur-free shooting.

[0018] Based on the above situation, after photographing the surface of the structure, the image quality of the photographed image is determined. If there is a photographed image inappropriate for the inspector's inspection, the location to be re-photographed is identified, and a method to improve the image quality of the photographed image is identified, and re-photographing is carried out. Here, the determination of the image quality of the photographed image and the judgment on the necessity of re-photographing are carried out at the photographing site immediately after photographing from the perspective of cost suppression. For example, when the structure to be inspected is far from the inspector's office and adjustments such as borrowing photographing equipment are required, re-photographing the inspection target is difficult because a great deal of labor is generated. Generally, the determination of the image quality of the photographed image and the judgment on the necessity of re-photographing are carried out based on the inspector's experience and know-how, but it is difficult to appropriately carry out in a short time. Therefore, a technology to support the inspector's image quality determination and re-photographing judgment is required.

[0019] FIG. 1A is a diagram showing an example of the functional configuration of an information processing apparatus. The information processing apparatus 100 includes a photographing unit 101, a storage unit 102, a determination unit 103, a synthesis unit 104, and a notification unit 105.

[0020] The photographing unit 101 is an imaging device that photographs a subject, and includes, for example, cameras mounted on a moving body (such as a drone) and a pan-tilt head. The photographing unit 101 images the surface of the structure to be inspected (referred to as the inspection target surface) and generates photographed image data. The "photographed image data" includes image data of the inspection target, the model names of the imaging device and the lens, camera parameters such as the focal length, shutter speed, and ISO sensitivity at the time of photographing, the photographing distance information from the photographing unit 101 to the subject, and the focus degree information (defocus value) of the photographed image. The photographing unit 101 acquires the photographing distance information from the photographing unit 101 to the subject by known means such as a distance measuring device provided in the photographing unit 101.

[0021] The imaging unit 101 has, for example, two photoelectric conversion units in which a sensor included in the imaging unit 101 performs photoelectric conversion for each pixel. The imaging unit 101 calculates the focus information (defocus value) of the captured image based on the phase difference between two images recorded by each photoelectric conversion unit. The focus information is expressed as a value of 0.0 or more for each pixel, for example. When the focus information is 0.0, it means that there is no phase difference between the two images of that pixel and the focus is in focus. When the focus information is 0.0 or more, it means that there is a phase difference corresponding to the numerical value between the two images of that pixel and the focus is out of focus.

[0022] The mobile body (drone) or pan-tilt head equipped with the imaging unit 101 may be controlled by either user operation or automatic operation. The imaging unit 101 captures the inspection target surface comprehensively while changing the imaging range to the subject and acquires a plurality of captured images, regardless of whether it is user operation or automatic operation control. The imaging unit 101 captures images in which the imaging ranges for each inspection target surface overlap (duplicate) with each other in order to cause the synthesis unit 104 to synthesize the captured images.

[0023] For example, the imaging unit 101 may capture images such that the imaging ranges overlap (duplicate) with each other by repeating still image capture. Alternatively, the imaging unit 101 may obtain captured image data such that the imaging ranges overlap with each other by extracting (capturing) still images from the video after performing video capture. Here, the timing of still image capture and the timing of extracting still images from the video may be arbitrary timings according to user settings or may be timings automatically set by the imaging unit 101. The imaging unit 101 transmits the acquired plurality of captured image data to the storage unit 102.

[0024] The storage unit 102 is a storage device that stores various types of data within the information processing device 100, and includes, for example, an HDD, SSD, RAM, and ROM. The storage unit 102 receives captured image data from the shooting unit 101 and stores the captured image data in various storage media. The storage unit 102 transmits the captured image data to the determination unit 103 at the instruction of the CPU (not shown) of the information processing device 100. The storage unit 102 receives determination information (corresponding to the determination result) in which the determination unit 103 has determined whether or not the image quality of the captured image data is good. Based on the determination information, the storage unit 102 transmits only the captured image data with good image quality to the synthesis unit 104 at the instruction of the CPU (not shown).

[0025] The determination unit 103 receives captured image data from the storage unit 102 and determines the image quality of the captured image data. The determination unit 103 transmits the determination information, which determines the image quality of the captured image data, to the storage unit 102 and the notification unit 105. Based on the shooting distance from the shooting unit 101 to the subject (surface to be inspected), the camera's sensor size, focal length, and number of pixels, the determination unit 103 calculates the shooting resolution from the image quality information using the following equation 1. The image quality information includes, for example, shooting resolution, focus, and blur.

[0026]

number

[0027] The shooting distance and focal length are information recorded in the captured image data. The sensor size and number of pixels are unique values ​​for each model of imaging device (camera). Therefore, the determination unit 103 obtains unique values ​​by referring to a database stored in the storage unit 102, etc., from the model information of the imaging device (camera) recorded in the captured image data. The determination unit 103 determines the image quality of the captured image data by comparing the shooting resolution calculated based on equation 1 with a reference shooting resolution stored in the storage unit 102, etc., in advance. That is, the determination unit 103 determines that the image quality of the captured image data is good if the value of the shooting resolution is equal to or greater than the value of the reference shooting resolution. On the other hand, the determination unit 103 determines that the image quality of the captured image data is not good if the value of the shooting resolution is lower than the value of the reference shooting resolution.

[0028] Next, an example will be described in which the determination unit 103 uses "focus" from the image quality information to determine whether the image quality of the captured image data is good or bad. The determination unit 103 determines whether the image quality is good or bad based on the focus information recorded in the captured image data. The captured image data, for example, contains focus information values ​​for each pixel in the captured image. The focus information for each pixel is as described above, but if the value is 0.0, it indicates that the image is in focus. If the value is greater than 0.0, it indicates that the image is out of focus according to that value.

[0029] The determination unit 103 determines the quality of focus for each captured image based on the extent to which pixels with a predetermined value or less are included in the entire captured image. For example, the determination unit 103 determines that the focus of the captured image is good if pixels with a focus information value of 1.0 or less account for 50% or more of the entire captured image. On the other hand, the determination unit 103 determines that the focus of the captured image is not good if pixels with a focus information value of 1.0 or less account for less than 50% of the entire captured image.

[0030] Next, we will explain an example in which the determination unit 103 uses "blur" from the image quality information to determine whether the image quality of the captured image data is good or bad. The determination unit 103 determines the presence or absence of blur by performing frequency analysis on the image data recorded in the captured image data. If the captured image data contains blur, fine cracks etc. will be visible over a wide area in the captured image, resulting in captured image data with low frequencies. Therefore, the determination unit 103 calculates the spatial frequency for each region of the captured image using a known method such as Fourier transform. The determination unit 103 determines the presence or absence of blur in the captured image data by using the average value obtained by averaging the frequency components calculated for each region on the captured image. That is, if the average value shows a high frequency, the determination unit 103 determines that the captured image data does not contain blur. On the other hand, if the average value shows a low frequency, the determination unit 103 determines that the captured image data contains blur.

[0031] The determination unit 103 determines the image quality of the captured image data from the storage unit 102, generates "determination information" indicating whether the image quality of the captured image data is good or not, and transmits it to the storage unit 102 and the notification unit 105. Here, if the determination unit 103 determines one element of image quality information (one of the following: shooting resolution, focus, or blur) from the captured image data, the determination result for that one element becomes the determination information. If the determination unit 103 determines multiple elements of image quality information from the captured image data, the determination result based on the combination of multiple elements becomes the determination information.

[0032] Figure 1B shows an example of judgment information (judgment result) obtained when the judgment unit 103 determines the image quality of captured image data. In the judgment information table 110, ○ indicates good image quality, and × indicates poor image quality. The judgment information 110 includes a number indicating the order of judgment, shooting resolution, focus, blur, and image quality judgment. For example, if the judgment unit 103 determines that all elements of the image quality information (shooting resolution, focus, blur) are good, it generates judgment information (judgment result) indicating that the image quality of the captured image data is good. Specifically, in the case of "number=1", if the shooting resolution is ○, the focus is ○, and the blur is ○, the judgment unit 103 determines the image quality to be ○.

[0033] On the other hand, if the determination unit 103 determines that any one element of the image quality information (for example, blur) is not good, it generates determination information 110 indicating that the image quality of the captured image data is not good. Specifically, in "number=2", if the shooting resolution is ○, the focus is ○, and the blur is ×, the determination unit 103 determines that the image quality is ×. Alternatively, the determination unit 103 may generate determination information indicating that the image quality of the captured image data is good if, for example, any one element of the image quality information (for example, shooting resolution) is good. Specifically, in "number=3", if the shooting resolution is ○, the focus is ×, and the blur is ×, the determination unit 103 determines that the image quality is ○. The determination information 110 may also include the determination results for each element (shooting resolution, focus, blur) used to evaluate the image quality of the captured image data, as shown in Figure 1B.

[0034] The synthesis unit 104 receives captured image data from the storage unit 102 and synthesizes the captured image data using a known method to generate composite image data. For example, when the synthesis unit 104 synthesizes captured image data taken using a fixed panoramic head, it uses a panoramic synthesis method to synthesize the captured image data. Also, when the synthesis unit 104 synthesizes captured image data taken using a mobile device (drone), it uses a three-dimensional reconstruction method to synthesize the captured image data. When the synthesis unit 104 uses the above methods, it is not always possible to synthesize all of the captured image data. For example, when the synthesis unit 104 generates a composite image using captured image data with little overlap (duplication) with other captured image data, it may not be able to generate a composite image without missing pixels. Missing pixels refer to pixels for which no pixel value is recorded.

[0035] Figure 2 shows an example of a composite image generated by an information processing device. In Figure 2, the composite image 201 includes regions 202, 203, and 204. Region 202 is a region where a total of eight pixels are missing, and is referred to as "missing area A due to insufficient overlap (image duplication)." The compositing unit 104 may generate a composite image 201 that includes missing pixels, as shown in region 202 of the composite image 201.

[0036] Furthermore, the captured image data used to generate the composite image 201 is not all of the captured image data captured by the shooting unit 101, but only the captured image data that the determination unit 103 has determined to have good image quality. Therefore, areas (pixels) in the captured image data that have been determined to have poor image quality are not included in the generation of the composite image 201, and are displayed as area 203 in the composite image 201. Area 203 is an area in which a total of 13 pixels are missing, and is "missing area B due to poor image quality".

[0037] Furthermore, if the imaging unit 101 is unable to photograph the subject (surface to be inspected) for any reason, and the captured image data includes areas that were not photographed, these areas will be displayed as regions 204 on the composite image 201. These reasons include situations where the imaging unit 101 mounted on a high-speed moving object (drone) is unable to photograph the desired subject area, or where the shutter cannot be released due to poor contact between the imaging device and lens of the imaging unit 101. Region 204 is an area where one pixel is missing and is referred to as the "missing area C due to missed image capture."

[0038] As described above, after generating the composite image 201, the synthesis unit 104 identifies regions 202 to 204 on the composite image 201 where there is no recording of pixel values ​​(pixels are missing). Next, the synthesis unit 104 estimates the reason for the pixel loss in the order of region 204 (missing area C), region 203 (missing area B), and region 202 (missing area A). Note that the order in which the reasons for pixel loss are estimated is just one example and is not limited to this.

[0039] For example, the synthesis unit 104 identifies the "used captured image data" used when synthesizing the composite image 201 that constitutes the periphery of region 203 and region 204, and identifies used captured image data that is adjacent to the used captured image data in terms of shooting order. For example, it is assumed that there are no missing (missing) pixels in the used captured image data that constitutes the periphery of region 204 (missing area C due to missed shooting) and the used captured image data that is adjacent to it in terms of shooting order. For example, the synthesis unit 104 can determine whether or not there are missing (missing) pixels in the captured image data based on whether or not a unique number (e.g., serial number) assigned to each used captured image data exists. Therefore, if the synthesis unit 104 determines that there are no missing (missing) pixels in the used captured image data located around region 204, it estimates that the reason for the pixel loss in region 204 (missing area C) is "insufficient image". Here, "insufficient image" includes reasons for pixel loss due to "missed shooting" and "insufficient overlap". Furthermore, the synthesis unit 104 can determine, as described later, whether the reason for the pixel loss in region 204 (defect area C) is either "missing image capture" or "insufficient overlap."

[0040] On the other hand, if there are missing (missing) pixels in used image data that are adjacent to the used image data in the order of shooting, it means that there is image data that the synthesis unit 104 has not received from the storage unit 102. In other words, it is assumed that there are missing (missing) pixels in used image data that are adjacent to the used image data that constitutes the area surrounding region 203 (the area B with image quality defects). Therefore, if the synthesis unit 104 determines that there are missing (missing) pixels in used image data located around region 203, it estimates that the reason for the missing pixels in region 203 is "image quality defects".

[0041] Alternatively, the synthesis unit 104 may identify the captured image data received from the storage unit 102 that was not used to generate the synthesized image 201 (referred to as unused captured image data). The synthesis unit 104 identifies the captured image data used to generate the synthesized image 201 (referred to as used captured image data) whose shooting time is close to that of the unused captured image data (for example, the difference in shooting time is within a predetermined range). The synthesis unit 104 identifies which part of the synthesized image 201 the used captured image data constitutes (i.e., the position of the used captured image data on the synthesized image 201). For example, the used captured image data that surrounds region 203 (missing area B due to poor image quality) of the synthesized image 201 is expected to have a shooting time close to that of the unused captured image data. Therefore, if the synthesis unit 104 finds that the area of ​​missing pixels is located around the used captured image data identified on the synthesized image 201, it estimates that the reason for the missing pixels in region 203 is "poor image quality". Furthermore, the synthesis unit 104 can estimate that the reason for loss in areas where the reason for loss was not determined to be "poor image quality" (for example, area 204) is "insufficient image".

[0042] Furthermore, the synthesis unit 104 can determine whether the "image shortage" is caused by "insufficient overlap" between used images, or by "missed shots" where there is insufficient captured image data to be used in the synthesis of the composite image 201. The synthesis unit 104 determines whether the "image shortage" is due to "insufficient overlap" or "missed shots" using the following method.

[0043] First, the compositing unit 104 estimates the reason for pixel loss in region 202 using the estimation method used to estimate the reason for pixel loss in region 204 (missing area C) ("insufficient image"). As a result, the compositing unit 104 estimates that the reason for pixel loss in both region 202 and region 204 is "insufficient image". Furthermore, the compositing unit 104 determines whether the reason for pixel loss in each region is either "missed image capture" or "insufficient overlap" based on whether the number of missing pixels in each region 202 and region 204 (the size of the pixel loss area) exceeds a predetermined threshold.

[0044] The predetermined threshold is the number of missing pixels, for example, four. If the compositing unit 104 determines that the number of missing pixels (1) in region 204 of Figure 2 does not exceed the predetermined threshold (for example, four), it estimates that the reason for the missing pixels in region 204 is "missed image capture." On the other hand, if the compositing unit 104 determines that the number of missing pixels (8) in region 202 exceeds the predetermined threshold (for example, four), it estimates that the reason for the missing pixels in region 202 is "insufficient overlap."

[0045] Figure 2 shows the case where the number of missing pixels in region 202 (8 pixels) is greater than the number of missing pixels in region 204 (1 pixel). However, if the imaging unit 101 fails to capture the subject (there are not enough images available for the composite image 201), there is a high probability that pixel loss will occur over a wide area of ​​the composite image 201. In other words, the area of ​​missing pixels due to missed capture (region 204) will be larger than the area of ​​missing pixels due to insufficient overlap (region 202), so the number of missing pixels in region 204 can be greater than that in region 202. Therefore, the compositing unit 104 can determine whether the reason for the pixel loss ("insufficient image") is "missed capture" or "insufficient overlap" by setting a predetermined threshold considering the number of missing pixels.

[0046] An alternative estimation method for cases where the reason for pixel loss is "missed shooting" will be explained. The compositing unit 104 acquires the communication history (information including time and event, and shutter interval information) related to the contact failure between the imaging device (camera) and lens of the shooting unit 101 that caused the "missed shooting". Then, the compositing unit 104 identifies used images taken at shooting times adjacent to the time when the camera shutter did not fire (shooting failure event occurred). Based on this, the compositing unit 104 can estimate that the reason for pixel loss in region 204 is "missed shooting" based on whether or not there is a pixel loss region around the used image identified by the above method on the composite image 201. An alternative estimation method for cases where the reason for pixel loss is "insufficient overlap" will also be explained. The compositing unit 104 may acquire the time when the subject was photographed when the movement speed of the mobile body (drone) equipped with the shooting unit 101 exceeded a threshold, and identify used images taken at shooting times adjacent to the acquired time. As a result, the synthesis unit 104 can estimate that the reason for the pixel loss in region 202 is "insufficient overlap" based on whether or not there is a pixel loss region around the used image identified on the synthesized image 201 using the method described above. The threshold for the movement speed of the moving object (drone) may be any numerical value of the speed at which "insufficient overlap" occurs when the shooting unit 101 photographs the subject.

[0047] The synthesis unit 104 estimates the reason for the loss in regions 202 to 204, then associates each estimation result with the coordinates of regions 202 to 204 on the synthesized image 201 and records it in the synthesized image data. The synthesis unit 104 then transmits the generated synthesized image data to the notification unit 105.

[0048] The notification unit 105 notifies the camera unit 101 of the method for re-shooting the subject (surface to be inspected) based on the image quality determination information of the captured image data from the determination unit 103 and the composite image data from the composite unit 104. The composite image data records the coordinates of regions 202 to 204 in the composite image 201 and the reasons for the loss of regions 202 to 204. The notification unit 105 refers to the reasons for the loss of regions 202 to 204 from the composite image data and presents a re-shooting method corresponding to the reason for the loss, corresponding to the coordinates of regions 202 to 204 in the composite image 201.

[0049] Figure 3 shows an example of how the information processing device presents a re-shooting method. The notification unit 105 notifies various parts of the information processing device, the CPU (not shown), and the user's terminal (not shown) of the re-shooting method for the composite image 201 and regions 202-204 in Figure 3. The user's terminal includes, for example, a smartphone, tablet, or notebook PC.

[0050] If the reason for the missing area in region 202 is "insufficient overlap," the notification unit 105 notifies the camera unit 101 to readjust the speed at which it changes the shooting range when photographing the subject (surface to be inspected). For example, if the camera unit 101 is mounted on a mobile device (drone), the notification unit 105 notifies the camera unit to slow down the speed at which it moves. If the speed at which the shooting range for the subject (surface to be inspected) is changed is slowed down, the overlap between the captured image data will increase, thus resolving the "insufficient overlap" on the composite image 201.

[0051] The notification unit 105 may, when the mobile body (drone) equipped with the shooting unit 101 is moving autonomously, directly access the shooting unit 101 and change the settings to reduce the speed of the mobile body. When the mobile body (drone) equipped with the shooting unit 101 is being moved by user control, the notification unit 105 may notify the user's terminal (not shown) to reduce the speed of the mobile body. In addition, the synthesis unit 104 may generate a composite image based on a group of still images extracted from a video of the subject (surface to be inspected) captured by the shooting unit 101. In this case, if a region 202 due to "insufficient overlap" occurs in the composite image 201, the notification unit 105 will notify the user to shorten the interval for extracting (capturing) still images from the video.

[0052] If the reason for the missing area 204 is "missed shooting," the notification unit 105 notifies the user to change the interval at which the shutter of the shooting unit 101 is released. For example, if the shutter of the shooting unit 101 is controlled by user operation, the notification unit 105 notifies the user's terminal, etc., to shorten the interval between shutter operations. If the shutter of the shooting unit 101 is an interval shutter that is operated automatically, the notification unit 105 notifies the user to reset the shooting interval. Here, an interval shutter is a setting that "takes one still image every X seconds," so the notification unit 105 notifies the user to shorten "X seconds." The notification unit 105 may also directly access the shooting unit 101 and change the setting of the interval shutter. Furthermore, if the shutter of the shooting unit 101 is controlled by user operation, the notification unit 105 may notify the user to operate the shutter without making any mistakes.

[0053] If the reason for the missing area 203 is "poor image quality," the notification unit 105 notifies the camera to change the shooting settings of the camera unit 101. To change the shooting settings of the camera unit 101, the notification unit 105 uses the image quality judgment information of unused captured image data that was not received from the judgment unit 103. If the reason for the missing area 203 is "poor image quality," there is always captured image data in the judgment information that has image quality information that has been determined to be of poor quality. Image quality information that has been determined to be of poor quality indicates that one of the following is not good: shooting resolution, focus, or blur. The notification unit 105 refers to the image quality information that has poor image quality and determines what changes to the shooting settings of the camera unit 101 should be made.

[0054] For example, if the notification unit 105 determines from the image quality judgment information (judgment result) that the "shooting resolution" is not good, it will notify the system to change the focal length of the shooting unit 101 to the telephoto side. When the focal length is changed to the telephoto side, the shooting range becomes narrower, so the notification unit 105 will notify the system to slow down the speed at which the shooting range of the shooting unit 101 is changed, or to shorten the shooting interval of the shooting unit 101. Alternatively, the notification unit 105 may notify the system to keep the focal length of the shooting unit 101 the same and move the shooting position of the shooting unit 101 closer to the subject (surface to be inspected). This shortens the shooting distance of the shooting unit 101 to the subject (surface to be inspected), thereby increasing the shooting resolution.

[0055] If the notification unit 105 determines that the "focus" is not good in the image quality judgment information (judgment result), it notifies the camera unit 101 to stop down the F-number. This increases the depth of field, making it easier to focus on the subject (surface to be inspected).

[0056] If the notification unit 105 determines that the image quality judgment information (judgment result) indicates that "blur" is not good, it will notify the user to increase the shutter speed of the shooting unit 101. In this case, the notification unit 105 may also notify the user to increase the ISO sensitivity in addition to the above notification. Blur can also occur if the shooting range of the shooting unit 101 is changed too quickly. Therefore, the notification unit 105 may also notify the user to slow down the shooting range of the shooting unit 101. If the mobile body (drone) equipped with the shooting unit 101 is moving autonomously, the notification unit 105 may directly access the shooting unit 101 and slow down the speed of the mobile body. On the other hand, if the mobile body equipped with the shooting unit 101 is moved by user operation, the notification unit 105 may notify the user's terminal (not shown) to slow down the speed of the mobile body. In this case, when the notification unit 105 notifies the user's terminal of a change in the shooting settings of the shooting unit 101, it may directly access the shooting unit 101 and change the shooting settings on behalf of the user.

[0057] Figure 4 is a flowchart illustrating the processing flow by the information processing device. The flowchart in Figure 4 will be explained below with reference to Figures 1A to 3. The program code following the flowchart is stored in the memory of the information processing device 100's storage unit 102 (RAM, ROM, etc.). The CPU (not shown) reads and executes the program, thereby realizing the process. In this flowchart, data transmission and reception may be performed directly or via a network such as wired or wireless connections.

[0058] In S401, the imaging unit 101 takes a photograph of the subject (surface to be inspected) of the structure to be inspected and acquires image data. The imaging unit 101 transmits the image data to the storage unit 102, and processing proceeds to S402.

[0059] In S402, the memory unit 102 stores the captured image data. The memory unit 102 transmits the captured image data to the determination unit 103, and the processing proceeds to S403.

[0060] In S403, the determination unit 103 determines the image quality of each of the multiple captured image data based on the image quality information of each image data. The determination unit 103 transmits the determination information (determination result) of the determined image quality of each captured image data to the storage unit 102 and the notification unit 105, and the process proceeds to S404.

[0061] In S404, the storage unit 102 selects the image data to generate a composite image based on the judgment information (judgment result) of each image data. That is, the storage unit 102 selects only the image data that has good image quality according to the image quality judgment information (judgment result) from among the image data. The storage unit 102 sends the selected image data to the composite unit 104, and the process proceeds to S405.

[0062] In S405, the synthesis unit 104 combines the captured image data with good image quality using a known method (for example, a panoramic synthesis method or a 3D reconstruction method) to create a composite image 201, and the processing proceeds to S406.

[0063] In S406, the compositing unit 104 determines whether or not to change the shooting settings of the shooting unit 101 based on whether or not there are areas of missing pixels (no recorded pixel values) in the composite image 201. If the compositing unit 104 determines that there are areas of missing pixels (no recorded pixel values) in the composite image 201 (Yes in S406), the process proceeds to S407. If the compositing unit 104 determines that there are no areas of missing pixels (no recorded pixel values) in the composite image 201 (No in S406), the process ends.

[0064] In S407, the synthesis unit 104 determines the regions 202-204 on the synthesized image 201 where there are missing pixels (no recorded pixel values) and estimates the reason for the missing pixels in regions 202-204. The synthesis unit 104 associates the reason for the missing pixels with each position (coordinate) in regions 202-204 and records it in the synthesized image data. The synthesis unit 104 transmits the synthesized image data to the notification unit 105. The notification unit 105 notifies the camera of the method for reshooting using the camera unit 101, based on the image quality determination information of the captured image data from the determination unit 103 and the synthesized image 201 from the synthesis unit 104. The process then proceeds to S408.

[0065] In S408, the notification unit 105 changes various settings related to the re-shooting of the subject (surface to be inspected) corresponding to areas 202-204 by the shooting unit 101, based on user operation or selection of a re-shooting method by the notification unit 105, and the process returns to S401.

[0066] As described above, according to the first embodiment, the image quality is determined based on the image quality information of the captured image data, and a composite image is generated using an image with good image quality from the image quality determination information (determination result). According to the first embodiment, if there are pixel loss areas in the composite image, a method for re-shooting the pixel loss areas can be estimated from the relationship between the shooting times of the used image and the unused image. As a result, if there are pixel loss areas in the composite image, the pixel loss areas can be visualized as re-shoot locations, and a method for re-shooting the subject corresponding to the pixel loss areas can be presented.

[0067] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0068] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0069] 100: Information processing unit, 101: Imaging unit, 102: Storage unit, 103: Judgment unit, 104: Synthesis unit, 105: Notification unit

Claims

1. a storage means for storing image quality information of each of a plurality of images showing a subject; a determining means for determining whether the image quality of the image is good or not based on the image quality information of the image; a synthesis means for synthesizing a synthetic image using the images determined by the determination means to have good image quality; a notification means for notifying a method for re-photographing the subject corresponding to the pixel-deficient area when the pixel-deficient area is found on the composite image; An information processing device comprising:

2. further comprising an estimation means for estimating a reason for the pixel loss when the pixel loss area is present on the composite image; 2. The information processing apparatus according to claim 1, wherein:

3. the estimation means estimates a position on the composite image of an already-used image used in composing the composite image, the difference in shooting time between the used image and an unused image not used in composing the composite image being within a predetermined range, and estimates a reason for the pixel loss based on whether or not there is a pixel loss area around the used image.

3. The information processing apparatus according to claim 2, wherein:

4. the estimation means estimates the reason for the pixel loss based on whether or not there is a loss between an image located around the pixel loss area on the composite image and an image adjacent to the image in the photographing order among images used when combining the composite image; 4. The information processing apparatus according to claim 2, wherein the information processing apparatus is a computer.

5. the estimation means estimates the reason for the pixel loss based on whether or not the size of the pixel loss area on the composite image exceeds a threshold.

5. The information processing device according to claim 2, wherein the information processing device is a computer.

6. the estimation means estimates a position on the composite image of a used image where the difference between the time and the photographing time is within a predetermined range, based on at least one of the time of occurrence of a photographing failure event in the photographing means that photographs the subject and the time when the moving speed of the mobile body equipped with the photographing means exceeds a threshold, and estimates the reason for the pixel loss based on whether or not there is a pixel loss area around the used image.

6. The information processing device according to claim 2, wherein the information processing device is a computer.

7. the notification means notifies a method for re-photographing the subject corresponding to the pixel-defective region based on the reason for the pixel-defective region estimated by the estimation means.

7. The information processing device according to claim 2, wherein the information processing device is a computer.

8. The reason for the pixel loss includes at least one of insufficient overlap, missing images, and poor image quality.

8. The information processing device according to claim 2, wherein the information processing device is a computer.

9. the notification means notifies the user of a method for re-photographing the subject corresponding to the pixel-deficient region based on image quality information of an unused image that was not used when synthesizing the composite image.

9. The information processing device according to claim 1, wherein the information processing device is a computer.

10. the method for re-photographing the object corresponding to the pixel-deficient region includes a method for changing at least one of a speed at which a photographing range is changed when photographing the object and a photographing setting when photographing the object, 10. The information processing device according to claim 1, wherein the information processing device is a computer.

11. the notification means notifies the user's terminal of information associating the position of the pixel-deficient area on the composite image with a method of re-photographing the subject corresponding to the pixel-deficient area.

11. The information processing device according to claim 1,

12. the image quality information of the image includes a plurality of factors for evaluating the image quality of the image; The determining means determines whether the image quality of the image is good based on at least one of the plurality of factors.

12. The information processing device according to claim 1, wherein the information processing device is a computer.

13. the determining means determines whether the image quality of the image is good based on at least one of the photographing resolution, focus, and blur.

13. The information processing device according to claim 1, wherein the information processing device is a computer.

14. The storage means stores the determination result made by the determination means, and transmits the image whose image quality is indicated by the determination result to the synthesis means.

14. The information processing device according to claim 1,

15. Further comprising an imaging means for imaging the subject.

15. The information processing device according to claim 1,

16. The subject is a structure, and the information processing device is used to inspect the structure.

16. The information processing device according to claim 1,

17. a moving body including an imaging means for imaging the subject; and an information processing device according to any one of claims 1 to 16. An imaging system characterized by:

18. With a tripod head, an imaging means mounted on the platform for imaging the subject; and an information processing device according to any one of claims 1 to 16. An imaging system characterized by:

19. A method executed by an information processing device, a determining step of determining whether the image quality of each of a plurality of images showing a subject is good based on image quality information of the images; a synthesis step of synthesizing a composite image using the images determined to have good image quality in the determination step; a notification step of notifying a method for re-photographing the subject corresponding to the pixel-deficient area when the pixel-deficient area is found on the composite image; A method for providing

20. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 16.