Information processing device, imaging system, method, and program

Abstract

To efficiently capture a subject that needs to be re-captured.SOLUTION: An information processing device stores image quality information for each of a plurality of images in which a subject is captured, determines whether the image quality of the images is good or not on the basis of the image quality information of the images, synthesizes a synthesized image using an image whose image quality is determined to be good, and notifies how to recapture the subject corresponding to a pixel-deficient region when a region of missing pixels on the synthesized image exists.SELECTED DRAWING: Figure 1A

Description

【Technical Field】 【0001】 The present invention relates to an information processing apparatus, a photographing system, a method, and a program. 【Background Art】 【0002】 An inspector of structures such as bridges and tunnels checks the position and degree of deformation (such as cracks and water leakage) on the surface of the structure and records the inspection results. Generally, a civil engineer visually checks the presence or absence of deformation on the surface of the structure and records the position and size of the deformation in a notebook or the like. However, in recent years, due to the aging of civil engineers and the like, there is a problem of a shortage of civil engineers for inspecting structures. To address this problem, "image-based inspection" is being carried out, in which the surface of a structure is photographed in high definition with an imaging device (camera), and deformation is confirmed and recorded from the photographed image by image analysis technology. 【0003】 In image-based inspection, a lens-exchangeable high-pixel and high-definition imaging device (camera) is mounted on an automatic pan-tilt and a drone, etc., and the surface of the structure to be inspected is comprehensively photographed using the imaging device. When photographing the surface of a structure such as a bridge and a tunnel at a resolution capable of detecting deformation (cracks), dozens to hundreds of photographed images are obtained. However, since only a narrow range of the structure is shown in each photographed image, the inspector cannot grasp which range of the structure was photographed from each photographed image. Therefore, a large-scale stitched image (composite image) showing a wider range of the surface of the structure is created by stitching (compositing) each photographed image. As a result, the photographed images can be combined by overlapping the respective photographed ranges at the time of photographing. 【0004】 Patent Document 1 discloses a technique for determining an abnormality such as a defect in a photographed image taken by a photographing device mounted on an unmanned aircraft and specifying an image that requires re-photographing according to the presence or absence of the abnormality. Specifically, an image to be re-photographed is specified by detecting an abnormality in data such as a defect in image data during communication of the photographed image. 【Prior Art Documents】 [Patent Documents] 【0005】 [Patent Document 1] Patent No. 6619761 [Overview of the project] [Problems that the invention aims to solve] 【0006】 However, Patent Document 1 does not provide a method for identifying a re-captured image based on the image quality of the captured image, nor a method for re-captured a re-captured image. 【0007】 Thus, there is a challenge in efficiently photographing subjects that require reshoots. 【0008】 The present invention aims to efficiently photograph subjects that require re-photography. [Means for solving the problem] 【0009】 To achieve the object of the present invention, an information processing device according to one embodiment of the present invention has the following configuration. That is, the information processing device includes a storage means for storing image quality information for each of a plurality of images in which a subject is depicted, A determination means for determining whether the image quality is good or not based on the image quality information of the image, which is the shooting resolution, focus, and blur. A synthesis means for synthesizing a composite image using an image that has been determined to have good image quality by the aforementioned determination means, If there are missing pixel regions in the composite image, the estimation means estimates the reason for the missing pixels. On the aforementioned composite image The aforementioned If there is a missing pixel region, a notification means for notifying a method for re-photographing the subject corresponding to the missing pixel region, Equipped with, The estimation means estimates the reason for the pixel loss based on whether the size of the pixel loss region on the composite image exceeds a threshold. The notification means is, (1) If the determination means determines that the shooting resolution is not good based on the image quality information, it will notify the camera to change the focal length of the shooting means to the telephoto side. (2) If the determination means determines that the focus is not good based on the image quality information, it will notify the camera to reduce the F-number of the camera. (3) If the determination means determines that the blur is not good based on the image quality information, it will notify the shooting means to increase the shutter speed. (4) If the reason for the missing pixels is insufficient overlap, the shooting means will be notified to readjust the speed at which the shooting range is changed when shooting the subject. (5) If the reason for the missing pixels is that the image was not captured, the camera will notify the camera to shorten the interval between shutter releases. It is characterized by the following: [Effects of the Invention] 【0010】 According to the present invention, subjects that require re-photography can be photographed efficiently. [Brief explanation of the drawing] 【0011】 [Figure 1A] A diagram showing an example of the configuration of functions in an information processing device. [Figure 1B] This figure shows an example of judgment information used to determine the image quality of captured image data. [Figure 2] A diagram showing an example of a composite image created by an information processing device. [Figure 3] A diagram showing an example of a re-imaging method presented by an information processing device. [Figure 4] A flowchart illustrating the processing flow by an information processing device. [Modes for carrying out the invention] 【0012】 Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted. 【0013】 (First Embodiment) The information processing device stores the image quality information of each of a plurality of images in which a subject is reflected, and determines whether the image quality of the image is good based on the image quality information of the image. The information processing device synthesizes a composite image using the images determined to have good image quality, and if there is a pixel defect area on the composite image, notifies a method for re-photographing the subject corresponding to the pixel defect area. Note that the information processing device is mounted on a mobile body (drone) or a pan-tilt head, but may be mounted on all devices capable of mounting the information processing device. Further, the mobile body (drone) or the pan-tilt head may not be equipped with the information processing device, and a separately prepared information processing device may control the mobile body (drone) or the pan-tilt head by wireless communication or the like. Further, data may be exchanged between the mobile body (drone) or the pan-tilt head and the information processing device via a detachable storage medium. In this case, it becomes a photographing system including the information processing device and the mobile body (drone). Here, the image quality information refers to a plurality of elements (for example, shooting resolution, focus, and blur) for evaluating the image quality of an image obtained by shooting a subject. Image quality refers to the quality of an image obtained by shooting a subject. Whether the image quality is good or not is determined based on whether at least any one of the plurality of elements of the image quality information is good. The composite image is one image obtained by synthesizing a plurality of images obtained by shooting a subject. 【0014】 The following explains why technology is needed to support inspectors in judging image quality and deciding whether to retake images. For example, when inspecting extremely fine cracks with a width of 0.2 mm on the surface of a structure, if the imaging device (camera) is not set up properly, the extremely fine cracks may not be visible in the image. In particular, the shooting resolution (actual size corresponding to one pixel of the image), focus, and blur are important for ensuring image quality. When an inspector inspects cracks with a width of 0.2 mm or more, a shooting resolution of 0.5 mm / pixel or higher is sufficient. When an inspector inspects cracks with a width of 0.05 mm or more, a shooting resolution of 0.3 mm / pixel or higher is sufficient. It is possible to photograph cracks narrower than the shooting resolution, but to photograph even narrower cracks, it is necessary to photograph the cracks at a higher shooting resolution. For example, in bridge inspections under national jurisdiction, cracks wider than 0.05 mm are subject to inspection, and it is recommended to use a resolution higher than 0.3 mm / pixel for imaging (Ministry of Land, Infrastructure, Transport and Tourism (March 2021, Draft Manual for the Delivery of 3D Deliverables Using Inspection Support Technology (Image Measurement Technology))). 【0015】 Focus adjustment significantly affects how cracks appear in captured images. For example, in images of cracks taken out of focus, very fine cracks will not be visible due to blurring. In image-based inspection, since the entire surface of the structure (hereinafter referred to as the inspection surface) must be captured in the image, it is necessary to set the imaging device so that the entire image is in focus. In other words, the aperture (F-number) must be set so that the depth of field (the range in focus) includes the nearest point to the furthest point of the inspection surface. Increasing the F-number increases the depth of field and widens the range in focus. However, increasing the F-number slows down the shutter speed, making the subject (cracks) more prone to blurring. 【0016】 Also, if the F value is extremely increased, pinhole blur called "small aperture blur" occurs, which affects the appearance of cracks. Also, camera shake during imaging when photographing cracks affects the appearance of cracks. In a blurry image, cracks are difficult to distinguish from dirt on the surface of the structure. By increasing the ISO sensitivity, the shutter speed of the imaging device (camera) can be increased, but if the ISO sensitivity is increased too much, ISO noise will occur. Since the ISO sensitivity affects the appearance of cracks in the photographed image, it is necessary to adjust the shooting conditions so that the image can be taken without blur without increasing the ISO sensitivity value as much as possible. 【0017】 As described above, the points to be noted in photographing during inspection of structures are clear, but the optimal setting values vary depending on the shooting situation and are not predetermined. For example, the focal length to achieve the required shooting resolution changes according to the distance from the imaging device (camera) to the shooting object (crack). Since the required depth of field changes depending on the orientation of the shooting object surface (the surface of the crack), the optimal F value is different. Since the brightness around the shooting object (crack) also changes depending on the weather, the shutter speed required to take a non-blurry image also changes. For example, when mounting an imaging device (camera) on a drone to photograph the surface of a structure, the distance between the imaging device (camera) and the shooting object and the orientation of the shooting object surface may change immediately before shooting depending on the wind situation during shooting. Also, due to sudden gusts of wind during shooting, the optimal value of the shutter speed for taking a non-blurry image changes. 【0018】 Based on the above circumstances, after photographing the surface of the structure, the image quality is judged, and if there are any images unsuitable for the inspector's inspection, the areas that need to be re-photographed are identified, and methods to improve the image quality are identified and re-photographed. Here, the judgment of image quality and the decision of whether or not to re-photograph are made immediately at the shooting site from the perspective of controlling costs. For example, if the structure to be inspected is far from the inspector's office, or if arrangements such as borrowing photographic equipment are necessary, re-photographing the structure to be inspected is difficult as it would require a great deal of effort. Generally, the judgment of image quality and the decision of whether or not to re-photograph are made based on the inspector's experience and know-how, but it is difficult to do so appropriately in a short amount of time. Therefore, there is a need for technology that supports inspectors in judging image quality and deciding whether or not to re-photograph. 【0019】 Figure 1A shows an example of the configuration of the functions of an information processing device. The information processing device 100 includes an imaging unit 101, a storage unit 102, a determination unit 103, a synthesis unit 104, and a notification unit 105. 【0020】 The imaging unit 101 is an imaging device that photographs a subject, and includes, for example, a camera mounted on a mobile device (drone, etc.) and a pan / tilt head. The imaging unit 101 photographs the surface of the structure to be inspected (referred to as the inspection surface) and generates captured image data. The "captured image data" includes image data of the object to be inspected, the model names of the imaging device and lens, camera parameters such as focal length, shutter speed, and ISO sensitivity at the time of shooting, shooting distance information from the imaging unit 101 to the subject, and focus information (defocus value) of the captured image. The imaging unit 101 acquires the shooting distance information from the imaging unit 101 to the subject by known means, for example, a distance measuring device provided in the imaging unit 101. 【0021】 The imaging unit 101 has, for example, two photoelectric conversion units that perform photoelectric conversion for each pixel of the sensor equipped in the imaging unit 101. The imaging unit 101 calculates the degree of focus information (defocus value) of the captured image based on the phase difference between the two images recorded by each photoelectric conversion unit. The degree of focus information is expressed, for example, as a value of 0.0 or greater for each pixel. If the degree of focus information is 0.0, it means that there is no phase difference between the two images of that pixel and that it is in focus. If the degree of focus information is 0.0 or greater, it means that there is a phase difference between the two images of that pixel according to the numerical value and that it is out of focus. 【0022】 The mobile device (drone) or pan / tilt mount on which the imaging unit 101 is mounted may be controlled by either user operation or automatic operation. Regardless of whether it is controlled by user operation or automatic operation, the imaging unit 101 comprehensively photographs the inspection target surface while changing the shooting range for the subject, and acquires multiple images. The imaging unit 101 takes images in which the shooting ranges for each inspection target surface overlap (overlap) with each other in order to allow the image synthesis unit 104 to synthesize the images. 【0023】 For example, the shooting unit 101 may repeatedly take still images so that each shooting range overlaps with the others. Alternatively, the shooting unit 101 may take video footage and then extract (capture) still images from the video to obtain image data in which each shooting range overlaps with the others. Here, the timing of taking still images and the timing of extracting still images from video may be arbitrary timings set by the user, or they may be timings automatically set by the shooting unit 101. The shooting unit 101 transmits the multiple acquired image data to the storage unit 102. 【0024】 The storage unit 102 is a storage device that stores various types of data within the information processing device 100, and includes, for example, an HDD, SSD, RAM, and ROM. The storage unit 102 receives captured image data from the shooting unit 101 and stores the captured image data in various storage media. The storage unit 102 transmits the captured image data to the determination unit 103 at the instruction of the CPU (not shown) of the information processing device 100. The storage unit 102 receives determination information (corresponding to the determination result) in which the determination unit 103 has determined whether or not the image quality of the captured image data is good. Based on the determination information, the storage unit 102 transmits only the captured image data with good image quality to the synthesis unit 104 at the instruction of the CPU (not shown). 【0025】 The determination unit 103 receives captured image data from the storage unit 102 and determines the image quality of the captured image data. The determination unit 103 transmits the determination information, which determines the image quality of the captured image data, to the storage unit 102 and the notification unit 105. Based on the shooting distance from the shooting unit 101 to the subject (surface to be inspected), the camera's sensor size, focal length, and number of pixels, the determination unit 103 calculates the shooting resolution from the image quality information using the following equation 1. The image quality information includes, for example, shooting resolution, focus, and blur. 【0026】 【number】 【0027】 The shooting distance and focal length are information recorded in the captured image data. The sensor size and number of pixels are unique values ​​for each model of imaging device (camera). Therefore, the determination unit 103 obtains unique values ​​by referring to a database stored in the storage unit 102, etc., from the model information of the imaging device (camera) recorded in the captured image data. The determination unit 103 determines the image quality of the captured image data by comparing the shooting resolution calculated based on equation 1 with a reference shooting resolution stored in the storage unit 102, etc., in advance. That is, the determination unit 103 determines that the image quality of the captured image data is good if the value of the shooting resolution is equal to or greater than the value of the reference shooting resolution. On the other hand, the determination unit 103 determines that the image quality of the captured image data is not good if the value of the shooting resolution is lower than the value of the reference shooting resolution. 【0028】 Next, an example will be described in which the determination unit 103 uses "focus" from the image quality information to determine whether the image quality of the captured image data is good or bad. The determination unit 103 determines whether the image quality is good or bad based on the focus information recorded in the captured image data. The captured image data, for example, contains focus information values ​​for each pixel in the captured image. The focus information for each pixel is as described above, but if the value is 0.0, it indicates that the image is in focus. If the value is greater than 0.0, it indicates that the image is out of focus according to that value. 【0029】 The determination unit 103 determines the quality of focus for each captured image based on the extent to which pixels with a predetermined value or less are included in the entire captured image. For example, the determination unit 103 determines that the focus of the captured image is good if pixels with a focus information value of 1.0 or less account for 50% or more of the entire captured image. On the other hand, the determination unit 103 determines that the focus of the captured image is not good if pixels with a focus information value of 1.0 or less account for less than 50% of the entire captured image. 【0030】 Next, we will explain an example in which the determination unit 103 uses "blur" from the image quality information to determine whether the image quality of the captured image data is good or bad. The determination unit 103 determines the presence or absence of blur by performing frequency analysis on the image data recorded in the captured image data. If the captured image data contains blur, fine cracks etc. will be visible over a wide area in the captured image, resulting in captured image data with low frequencies. Therefore, the determination unit 103 calculates the spatial frequency for each region of the captured image using a known method such as Fourier transform. The determination unit 103 determines the presence or absence of blur in the captured image data by using the average value obtained by averaging the frequency components calculated for each region on the captured image. That is, if the average value shows a high frequency, the determination unit 103 determines that the captured image data does not contain blur. On the other hand, if the average value shows a low frequency, the determination unit 103 determines that the captured image data contains blur. 【0031】 The determination unit 103 determines the image quality of the captured image data from the storage unit 102, generates "determination information" indicating whether the image quality of the captured image data is good or not, and transmits it to the storage unit 102 and the notification unit 105. Here, if the determination unit 103 determines one element of image quality information (one of the following: shooting resolution, focus, or blur) from the captured image data, the determination result for that one element becomes the determination information. If the determination unit 103 determines multiple elements of image quality information from the captured image data, the determination result based on the combination of multiple elements becomes the determination information. 【0032】 Figure 1B shows an example of judgment information (judgment result) obtained when the judgment unit 103 determines the image quality of captured image data. In the judgment information table 110, ○ indicates good image quality, and × indicates poor image quality. The judgment information 110 includes a number indicating the order of judgment, shooting resolution, focus, blur, and image quality judgment. For example, if the judgment unit 103 determines that all elements of the image quality information (shooting resolution, focus, blur) are good, it generates judgment information (judgment result) indicating that the image quality of the captured image data is good. Specifically, in the case of "number=1", if the shooting resolution is ○, the focus is ○, and the blur is ○, the judgment unit 103 determines the image quality to be ○. 【0033】 On the other hand, if the determination unit 103 determines that any one element of the image quality information (for example, blur) is not good, it generates determination information 110 indicating that the image quality of the captured image data is not good. Specifically, in "number=2", if the shooting resolution is ○, the focus is ○, and the blur is ×, the determination unit 103 determines that the image quality is ×. Alternatively, the determination unit 103 may generate determination information indicating that the image quality of the captured image data is good if, for example, any one element of the image quality information (for example, shooting resolution) is good. Specifically, in "number=3", if the shooting resolution is ○, the focus is ×, and the blur is ×, the determination unit 103 determines that the image quality is ○. The determination information 110 may also include the determination results for each element (shooting resolution, focus, blur) used to evaluate the image quality of the captured image data, as shown in Figure 1B. 【0034】 The synthesis unit 104 receives captured image data from the storage unit 102 and synthesizes the captured image data using a known method to generate composite image data. For example, when the synthesis unit 104 synthesizes captured image data taken using a fixed panoramic head, it uses a panoramic synthesis method to synthesize the captured image data. Also, when the synthesis unit 104 synthesizes captured image data taken using a mobile device (drone), it uses a three-dimensional reconstruction method to synthesize the captured image data. When the synthesis unit 104 uses the above methods, it is not always possible to synthesize all of the captured image data. For example, when the synthesis unit 104 generates a composite image using captured image data with little overlap (duplication) with other captured image data, it may not be able to generate a composite image without missing pixels. Missing pixels refer to pixels for which no pixel value is recorded. 【0035】 Figure 2 shows an example of a composite image generated by an information processing device. In Figure 2, the composite image 201 includes regions 202, 203, and 204. Region 202 is a region where a total of eight pixels are missing, and is referred to as "missing area A due to insufficient overlap (image duplication)." The compositing unit 104 may generate a composite image 201 that includes missing pixels, as shown in region 202 of the composite image 201. 【0036】 Furthermore, the captured image data used to generate the composite image 201 is not all of the captured image data captured by the shooting unit 101, but only the captured image data that the determination unit 103 has determined to have good image quality. Therefore, areas (pixels) in the captured image data that have been determined to have poor image quality are not included in the generation of the composite image 201, and are displayed as area 203 in the composite image 201. Area 203 is an area in which a total of 13 pixels are missing, and is "missing area B due to poor image quality". 【0037】 Furthermore, if the imaging unit 101 is unable to photograph the subject (surface to be inspected) for any reason, and the captured image data includes areas that were not photographed, these areas will be displayed as regions 204 on the composite image 201. These reasons include situations where the imaging unit 101 mounted on a high-speed moving object (drone) is unable to photograph the desired subject area, or where the shutter cannot be released due to poor contact between the imaging device and lens of the imaging unit 101. Region 204 is an area where one pixel is missing and is referred to as the "missing area C due to missed image capture." 【0038】 As described above, after generating the composite image 201, the synthesis unit 104 identifies regions 202 to 204 on the composite image 201 where there is no recording of pixel values ​​(pixels are missing). Next, the synthesis unit 104 estimates the reason for the pixel loss in the order of region 204 (missing area C), region 203 (missing area B), and region 202 (missing area A). Note that the order in which the reasons for pixel loss are estimated is just one example and is not limited to this. 【0039】 For example, the synthesis unit 104 identifies the "used captured image data" used when synthesizing the composite image 201 that constitutes the periphery of region 203 and region 204, and identifies used captured image data that is adjacent to the used captured image data in terms of shooting order. For example, it is assumed that there are no missing (missing) pixels in the used captured image data that constitutes the periphery of region 204 (missing area C due to missed shooting) and the used captured image data that is adjacent to it in terms of shooting order. For example, the synthesis unit 104 can determine whether or not there are missing (missing) pixels in the captured image data based on whether or not a unique number (e.g., serial number) assigned to each used captured image data exists. Therefore, if the synthesis unit 104 determines that there are no missing (missing) pixels in the used captured image data located around region 204, it estimates that the reason for the pixel loss in region 204 (missing area C) is "insufficient image". Here, "insufficient image" includes reasons for pixel loss due to "missed shooting" and "insufficient overlap". Furthermore, the synthesis unit 104 can determine, as described later, whether the reason for the pixel loss in region 204 (defect area C) is either "missing image capture" or "insufficient overlap." 【0040】 On the other hand, if there are missing (missing) pixels in used image data that are adjacent to the used image data in the order of shooting, it means that there is image data that the synthesis unit 104 has not received from the storage unit 102. In other words, it is assumed that there are missing (missing) pixels in used image data that are adjacent to the used image data that constitutes the area surrounding region 203 (the area B with image quality defects). Therefore, if the synthesis unit 104 determines that there are missing (missing) pixels in used image data located around region 203, it estimates that the reason for the missing pixels in region 203 is "image quality defects". 【0041】 Alternatively, the synthesis unit 104 may identify the captured image data received from the storage unit 102 that was not used to generate the synthesized image 201 (referred to as unused captured image data). The synthesis unit 104 identifies the captured image data used to generate the synthesized image 201 (referred to as used captured image data) whose shooting time is close to that of the unused captured image data (for example, the difference in shooting time is within a predetermined range). The synthesis unit 104 identifies which part of the synthesized image 201 the used captured image data constitutes (i.e., the position of the used captured image data on the synthesized image 201). For example, the used captured image data that surrounds region 203 (missing area B due to poor image quality) of the synthesized image 201 is expected to have a shooting time close to that of the unused captured image data. Therefore, if the synthesis unit 104 finds that the area of ​​missing pixels is located around the used captured image data identified on the synthesized image 201, it estimates that the reason for the missing pixels in region 203 is "poor image quality". Furthermore, the synthesis unit 104 can estimate that the reason for loss in areas where the reason for loss was not determined to be "poor image quality" (for example, area 204) is "insufficient image". 【0042】 Furthermore, the synthesis unit 104 can determine whether the "image shortage" is caused by "insufficient overlap" between used images, or by "missed shots" where there is insufficient captured image data to be used in the synthesis of the composite image 201. The synthesis unit 104 determines whether the "image shortage" is due to "insufficient overlap" or "missed shots" using the following method. 【0043】 First, the compositing unit 104 estimates the reason for pixel loss in region 202 using the estimation method used to estimate the reason for pixel loss in region 204 (missing area C) ("insufficient image"). As a result, the compositing unit 104 estimates that the reason for pixel loss in both region 202 and region 204 is "insufficient image". Furthermore, the compositing unit 104 determines whether the reason for pixel loss in each region is either "missed image capture" or "insufficient overlap" based on whether the number of missing pixels in each region 202 and region 204 (the size of the pixel loss area) exceeds a predetermined threshold. 【0044】 The predetermined threshold is the number of missing pixels, for example, four. If the compositing unit 104 determines that the number of missing pixels (1) in region 204 of Figure 2 does not exceed the predetermined threshold (for example, four), it estimates that the reason for the missing pixels in region 204 is "missed image capture." On the other hand, if the compositing unit 104 determines that the number of missing pixels (8) in region 202 exceeds the predetermined threshold (for example, four), it estimates that the reason for the missing pixels in region 202 is "insufficient overlap." 【0045】 Figure 2 shows the case where the number of missing pixels in region 202 (8 pixels) is greater than the number of missing pixels in region 204 (1 pixel). However, if the imaging unit 101 fails to capture the subject (there are not enough images available for the composite image 201), there is a high probability that pixel loss will occur over a wide area of ​​the composite image 201. In other words, the area of ​​missing pixels due to missed capture (region 204) will be larger than the area of ​​missing pixels due to insufficient overlap (region 202), so the number of missing pixels in region 204 can be greater than that in region 202. Therefore, the compositing unit 104 can determine whether the reason for the pixel loss ("insufficient image") is "missed capture" or "insufficient overlap" by setting a predetermined threshold considering the number of missing pixels. 【0046】 An alternative estimation method for cases where the reason for pixel loss is "missed shooting" will be explained. The compositing unit 104 acquires the communication history (information including time and event, and shutter interval information) related to the contact failure between the imaging device (camera) and lens of the shooting unit 101 that caused the "missed shooting". Then, the compositing unit 104 identifies used images taken at shooting times adjacent to the time when the camera shutter did not fire (shooting failure event occurred). Based on this, the compositing unit 104 can estimate that the reason for pixel loss in region 204 is "missed shooting" based on whether or not there is a pixel loss region around the used image identified by the above method on the composite image 201. An alternative estimation method for cases where the reason for pixel loss is "insufficient overlap" will also be explained. The compositing unit 104 may acquire the time when the subject was photographed when the movement speed of the mobile body (drone) equipped with the shooting unit 101 exceeded a threshold, and identify used images taken at shooting times adjacent to the acquired time. As a result, the synthesis unit 104 can estimate that the reason for the pixel loss in region 202 is "insufficient overlap" based on whether or not there is a pixel loss region around the used image identified on the synthesized image 201 using the method described above. The threshold for the movement speed of the moving object (drone) may be any numerical value of the speed at which "insufficient overlap" occurs when the shooting unit 101 photographs the subject. 【0047】 The synthesis unit 104 estimates the reason for the loss in regions 202 to 204, then associates each estimation result with the coordinates of regions 202 to 204 on the synthesized image 201 and records it in the synthesized image data. The synthesis unit 104 then transmits the generated synthesized image data to the notification unit 105. 【0048】 The notification unit 105 notifies the camera unit 101 of the method for re-shooting the subject (surface to be inspected) based on the image quality determination information of the captured image data from the determination unit 103 and the composite image data from the composite unit 104. The composite image data records the coordinates of regions 202 to 204 in the composite image 201 and the reasons for the loss of regions 202 to 204. The notification unit 105 refers to the reasons for the loss of regions 202 to 204 from the composite image data and presents a re-shooting method corresponding to the reason for the loss, corresponding to the coordinates of regions 202 to 204 in the composite image 201. 【0049】 Figure 3 shows an example of how the information processing device presents a re-shooting method. The notification unit 105 notifies various parts of the information processing device, the CPU (not shown), and the user's terminal (not shown) of the re-shooting method for the composite image 201 and regions 202-204 in Figure 3. The user's terminal includes, for example, a smartphone, tablet, or notebook PC. 【0050】 If the reason for the missing area in region 202 is "insufficient overlap," the notification unit 105 notifies the camera unit 101 to readjust the speed at which it changes the shooting range when photographing the subject (surface to be inspected). For example, if the camera unit 101 is mounted on a mobile device (drone), the notification unit 105 notifies the camera unit to slow down the speed at which it moves. If the speed at which the shooting range for the subject (surface to be inspected) is changed is slowed down, the overlap between the captured image data will increase, thus resolving the "insufficient overlap" on the composite image 201. 【0051】 The notification unit 105 may, when the mobile body (drone) equipped with the shooting unit 101 is moving autonomously, directly access the shooting unit 101 and change the settings to reduce the speed of the mobile body. When the mobile body (drone) equipped with the shooting unit 101 is being moved by user control, the notification unit 105 may notify the user's terminal (not shown) to reduce the speed of the mobile body. In addition, the synthesis unit 104 may generate a composite image based on a group of still images extracted from a video of the subject (surface to be inspected) captured by the shooting unit 101. In this case, if a region 202 due to "insufficient overlap" occurs in the composite image 201, the notification unit 105 will notify the user to shorten the interval for extracting (capturing) still images from the video. 【0052】 If the reason for the missing area 204 is "missed shooting," the notification unit 105 notifies the user to change the interval at which the shutter of the shooting unit 101 is released. For example, if the shutter of the shooting unit 101 is controlled by user operation, the notification unit 105 notifies the user's terminal, etc., to shorten the interval between shutter operations. If the shutter of the shooting unit 101 is an interval shutter that is operated automatically, the notification unit 105 notifies the user to reset the shooting interval. Here, an interval shutter is a setting that "takes one still image every X seconds," so the notification unit 105 notifies the user to shorten "X seconds." The notification unit 105 may also directly access the shooting unit 101 and change the setting of the interval shutter. Furthermore, if the shutter of the shooting unit 101 is controlled by user operation, the notification unit 105 may notify the user to operate the shutter without making any mistakes. 【0053】 If the reason for the missing area 203 is "poor image quality," the notification unit 105 notifies the camera to change the shooting settings of the camera unit 101. To change the shooting settings of the camera unit 101, the notification unit 105 uses the image quality judgment information of unused captured image data that was not received from the judgment unit 103. If the reason for the missing area 203 is "poor image quality," there is always captured image data in the judgment information that has image quality information that has been determined to be of poor quality. Image quality information that has been determined to be of poor quality indicates that one of the following is not good: shooting resolution, focus, or blur. The notification unit 105 refers to the image quality information that has poor image quality and determines what changes to the shooting settings of the camera unit 101 should be made. 【0054】 For example, if the notification unit 105 determines from the image quality judgment information (judgment result) that the "shooting resolution" is not good, it will notify the system to change the focal length of the shooting unit 101 to the telephoto side. When the focal length is changed to the telephoto side, the shooting range becomes narrower, so the notification unit 105 will notify the system to slow down the speed at which the shooting range of the shooting unit 101 is changed, or to shorten the shooting interval of the shooting unit 101. Alternatively, the notification unit 105 may notify the system to keep the focal length of the shooting unit 101 the same and move the shooting position of the shooting unit 101 closer to the subject (surface to be inspected). This shortens the shooting distance of the shooting unit 101 to the subject (surface to be inspected), thereby increasing the shooting resolution. 【0055】 If the notification unit 105 determines that the "focus" is not good in the image quality judgment information (judgment result), it notifies the camera unit 101 to stop down the F-number. This increases the depth of field, making it easier to focus on the subject (surface to be inspected). 【0056】 If the notification unit 105 determines that the image quality judgment information (judgment result) indicates that "blur" is not good, it will notify the user to increase the shutter speed of the shooting unit 101. In this case, the notification unit 105 may also notify the user to increase the ISO sensitivity in addition to the above notification. Blur can also occur if the shooting range of the shooting unit 101 is changed too quickly. Therefore, the notification unit 105 may also notify the user to slow down the shooting range of the shooting unit 101. If the mobile body (drone) equipped with the shooting unit 101 is moving autonomously, the notification unit 105 may directly access the shooting unit 101 and slow down the speed of the mobile body. On the other hand, if the mobile body equipped with the shooting unit 101 is moved by user operation, the notification unit 105 may notify the user's terminal (not shown) to slow down the speed of the mobile body. In this case, when the notification unit 105 notifies the user's terminal of a change in the shooting settings of the shooting unit 101, it may directly access the shooting unit 101 and change the shooting settings on behalf of the user. 【0057】 Figure 4 is a flowchart illustrating the processing flow by the information processing device. The flowchart in Figure 4 will be explained below with reference to Figures 1A to 3. The program code following the flowchart is stored in the memory of the information processing device 100's storage unit 102 (RAM, ROM, etc.). The CPU (not shown) reads and executes the program, thereby realizing the process. In this flowchart, data transmission and reception may be performed directly or via a network such as wired or wireless connections. 【0058】 In S401, the imaging unit 101 takes a photograph of the subject (surface to be inspected) of the structure to be inspected and acquires image data. The imaging unit 101 transmits the image data to the storage unit 102, and processing proceeds to S402. 【0059】 In S402, the memory unit 102 stores the captured image data. The memory unit 102 transmits the captured image data to the determination unit 103, and the processing proceeds to S403. 【0060】 In S403, the determination unit 103 determines the image quality of each of the multiple captured image data based on the image quality information of each image data. The determination unit 103 transmits the determination information (determination result) of the determined image quality of each captured image data to the storage unit 102 and the notification unit 105, and the process proceeds to S404. 【0061】 In S404, the storage unit 102 selects the image data to generate a composite image based on the judgment information (judgment result) of each image data. That is, the storage unit 102 selects only the image data that has good image quality according to the image quality judgment information (judgment result) from among the image data. The storage unit 102 sends the selected image data to the composite unit 104, and the process proceeds to S405. 【0062】 In S405, the synthesis unit 104 combines the captured image data with good image quality using a known method (for example, a panoramic synthesis method or a 3D reconstruction method) to create a composite image 201, and the processing proceeds to S406. 【0063】 In S406, the compositing unit 104 determines whether or not to change the shooting settings of the shooting unit 101 based on whether or not there are areas of missing pixels (no recorded pixel values) in the composite image 201. If the compositing unit 104 determines that there are areas of missing pixels (no recorded pixel values) in the composite image 201 (Yes in S406), the process proceeds to S407. If the compositing unit 104 determines that there are no areas of missing pixels (no recorded pixel values) in the composite image 201 (No in S406), the process ends. 【0064】 In S407, the synthesis unit 104 determines the regions 202-204 on the synthesized image 201 where there are missing pixels (no recorded pixel values) and estimates the reason for the missing pixels in regions 202-204. The synthesis unit 104 associates the reason for the missing pixels with each position (coordinate) in regions 202-204 and records it in the synthesized image data. The synthesis unit 104 transmits the synthesized image data to the notification unit 105. The notification unit 105 notifies the camera of the method for reshooting using the camera unit 101, based on the image quality determination information of the captured image data from the determination unit 103 and the synthesized image 201 from the synthesis unit 104. The process then proceeds to S408. 【0065】 In S408, the notification unit 105 changes various settings related to the re-shooting of the subject (surface to be inspected) corresponding to areas 202-204 by the shooting unit 101, based on user operation or selection of a re-shooting method by the notification unit 105, and the process returns to S401. 【0066】 As described above, according to the first embodiment, the image quality is determined based on the image quality information of the captured image data, and a composite image is generated using an image with good image quality from the image quality determination information (determination result). According to the first embodiment, if there are pixel loss areas in the composite image, a method for re-shooting the pixel loss areas can be estimated from the relationship between the shooting times of the used image and the unused image. As a result, if there are pixel loss areas in the composite image, the pixel loss areas can be visualized as re-shoot locations, and a method for re-shooting the subject corresponding to the pixel loss areas can be presented. 【0067】 (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. 【0068】 The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of symbols] 【0069】 100: Information processing unit, 101: Imaging unit, 102: Storage unit, 103: Judgment unit, 104: Synthesis unit, 105: Notification unit

Claims

[Claim 1] A storage means for storing image quality information for each of multiple images in which the subject is captured, A determination means for determining whether the image quality is good or not based on the image quality information of the image, which is the shooting resolution, focus, and blur. A synthesis means for synthesizing a composite image using an image that has been determined to have good image quality by the aforementioned determination means, If there are missing pixel regions in the composite image, the estimation means estimates the reason for the missing pixels. If there is a missing pixel region in the composite image, a notification means for notifying a method for re-photographing the subject corresponding to the missing pixel region, Equipped with, The estimation means estimates the reason for the pixel loss based on whether the size of the pixel loss region on the composite image exceeds a threshold. The notification means is, (1) If the determination means determines that the shooting resolution is not good based on the image quality information, it will notify the user to change the focal length of the shooting means to the telephoto side. (2) If the determination means determines that the focus is not good based on the image quality information, it will notify the camera to reduce the F-number of the camera. (3) If the determination means determines that the blur is not good based on the image quality information, it will notify the camera to increase the shutter speed of the shooting means. (4) If the reason for the missing pixels is insufficient overlap, the shooting means will notify the system to readjust the speed at which the shooting range is changed when shooting the subject. (5) If the reason for the missing pixels is that the image was not captured, the system will be notified to shorten the interval at which the shutter of the imaging means is released. An information processing device characterized by the following: [Claim 2] The estimation means estimates the position on the composite image of a used image used when composing the composite image, provided that the difference in shooting time between the used image and an unused image not used when composing the composite image is within a predetermined range, and estimates the reason for the pixel loss based on whether or not there is a pixel loss region around the used image. The information processing apparatus according to feature 1. [Claim 3] The estimation means estimates the reason for the pixel loss based on whether or not there is a loss between the image located around the missing area of ​​the pixel on the composite image and the used image used when compositing the composite image that is adjacent to the image in terms of the order of capture. The information processing apparatus according to claim 1 or 2. [Claim 4] The estimation means estimates the position of a used image on the composite image where the difference between the time of the failed shooting event in the shooting means that photographs the subject and the time of the shooting when the moving speed of the mobile body on which the shooting means is mounted exceeds a threshold, based on at least one of the above, and estimates the reason for the pixel loss based on whether or not there is a pixel loss region around the used image. The information processing apparatus according to any one of claims 1 to 3. [Claim 5] The notification means notifies a method for re-photographing the subject corresponding to the pixel defect region, based on the reason for the pixel defect estimated by the estimation means. The information processing apparatus according to any one of claims 1 to 4. [Claim 6] The reasons for the loss of the aforementioned pixels include at least one of the following: insufficient overlap, missed capture, or poor image quality. The information processing apparatus according to any one of claims 1 to 5. [Claim 7] The notification means notifies a method for re-photographing the subject corresponding to the pixel defect region, based on the image quality information of the unused image that was not used when synthesizing the composite image. The information processing apparatus according to any one of claims 1 to 6. [Claim 8] The notification means notifies the user's terminal of information relating the location of the pixel defect region on the composite image to a method for re-photographing the subject corresponding to the pixel defect region. The notification means presents one of the notifications (1), (2), (3), (4), or (5) for each missing pixel region of the composite image. The information processing apparatus according to any one of claims 1 to 7. [Claim 9] The image quality information of the aforementioned image includes a plurality of elements for evaluating the image quality, The determination means determines whether the image quality is good or not based on at least one of the plurality of elements. The information processing apparatus according to any one of claims 1 to 8. [Claim 10] The storage means stores the determination result determined by the determination means and transmits the image, which is indicated by the determination result to be of good quality, to the synthesis means. The information processing apparatus according to any one of claims 1 to 9. [Claim 11] The system further comprises the aforementioned photographing means for photographing the aforementioned subject. The information processing apparatus according to any one of claims 1 to 10. [Claim 12] The subject is a structure, and the information processing device is used for inspecting the structure. The information processing apparatus according to any one of claims 1 to 11. [Claim 13] A mobile body equipped with the aforementioned photographing means for photographing the aforementioned subject, The information processing apparatus comprises the information processing apparatus according to any one of claims 1 to 12. A shooting system characterized by the following features. [Claim 14] Tripod head and The aforementioned shooting means, which is mounted on the aforementioned tripod head and photographs the aforementioned subject, The information processing device comprises the information processing device according to any one of claims 1 to 12, A photographic system characterized by the following features. [Claim 15] A method executed by an information processing device, A determination step in which the image quality is determined to be good or not based on the image quality information of each of the multiple images in which the subject is captured, namely the shooting resolution, focus, and blur. A synthesis step is performed to synthesize a composite image using the image that was determined to have good image quality in the above determination step, If there are missing pixel regions in the composite image, the estimation step involves estimating the reason for the missing pixels. If there is a missing pixel region in the composite image, a notification step is made to notify the method for re-photographing the subject corresponding to the missing pixel region. Equipped with, In the estimation step, the reason for the pixel loss is estimated based on whether the size of the pixel loss region on the composite image exceeds a threshold. In the notification process, (1) If the determination step determines that the shooting resolution is not good based on the image quality information, a notification is given to change the focal length of the shooting means to the telephoto side. (2) If the determination step determines that the focus is not good based on the image quality information, the camera will notify the camera to reduce the F-number of the shooting means. (3) If the determination step determines that the blur is not good based on the image quality information, the camera will be notified to increase the shutter speed of the shooting means. (4) If the reason for the missing pixels is insufficient overlap, the shooting means will notify the system to readjust the speed at which the shooting range is changed when shooting the subject. (5) If the reason for the missing pixels is that the image was not captured, the system will be notified to shorten the interval at which the shutter of the imaging means is released. A method characterized by the following: [Claim 16] A program for causing a computer to function as one of the means of an information processing apparatus described in any one of claims 1 to 12.