Information processing device and method

Abstract

This aims to make it easier to recapture abnormal images during photogrammetry imaging. [Solution] The system performs a process that includes immediately detecting anomalies in the captured image upon generation of an image for generating three-dimensional shape information representing the three-dimensional shape of a 3D object, and storing the captured image and the result of the anomaly detection in association with each other. The result of the anomaly detection may be presented. In addition, guidance for re-capturing the image in which an anomaly was detected may be presented. This disclosure can be applied, for example, to information processing devices, imaging devices, electronic devices, information processing methods, imaging methods, or programs.

Description

Technical Field

[0005] , , , ,

[0001] The present disclosure relates to an information processing apparatus and method, and more particularly, to an information processing apparatus and method that enable easier retaking of abnormal images in photogrammetry imaging.

Background Art

[0002] Conventionally, as a method for 3D modeling of a 3D object having a three-dimensional shape, there has been a method called photogrammetry in which the 3D object is imaged from multiple directions and 3D data is generated based on a plurality of obtained imaging images. The imaging images used in such photogrammetry are required to have high quality in order to more accurately reproduce the reality of the subject. However, since a large number of imaging images are required to generate 3D data, in the imaging operation, it is required to perform a larger number of imaging in a shorter time. Furthermore, since it is necessary to repeat imaging while changing the viewpoint, it is difficult to perform imaging while paying attention to improving the quality of the image, and there is a high possibility that imaging images including excessive blur or shake may occur frequently.

[0003] In addition, there has been a method for detecting and notifying an abnormal image, which is such an imaging image including excessive blur or shake (see, for example, Patent Document 1 and Patent Document 2).

Prior Art Documents

[0006] This disclosure is made in light of these circumstances and aims to make it easier to recapture abnormal images in photogrammetry imaging. [Means for solving the problem]

[0007] One aspect of this technology is an information processing device comprising: an anomaly detection unit that immediately performs anomaly detection on an captured image when an captured image is generated to generate three-dimensional shape information representing the three-dimensional shape of a 3D object; and a storage unit that stores the captured image and the result of the anomaly detection in association with each other.

[0008] One aspect of this technology is an information processing method that includes, immediately upon generation of an image for generating three-dimensional shape information representing the three-dimensional shape of a 3D object, performing anomaly detection on the image and storing the image and the result of the anomaly detection in association.

[0009] In one aspect of this technology, the information processing device and method perform, immediately upon generation of an image for generating three-dimensional shape information representing the three-dimensional shape of a 3D object, anomaly detection is performed on the image, and the image and the result of the anomaly detection are stored in association with each other. [Brief explanation of the drawing]

[0010] [Figure 1] This is a diagram illustrating the basics of photogrammetry. [Figure 2] This figure shows an example of a method for detecting abnormal images. [Figure 3] This figure shows an example of a method for detecting abnormal images. [Figure 4] This is a diagram showing an example of the state of immediate presentation of the abnormality detection result. [Figure 5] This is a diagram showing an example of the state of immediate presentation of the abnormality detection result. [Figure 6] This is a diagram showing an example of the state of presentation after the imaging operation of the abnormality detection result. [Figure 7] This is a diagram showing an example of the state of presentation after the imaging operation of the abnormality detection result. [Figure 8] This is a diagram showing an example of the state of presentation after the imaging operation of the abnormality detection result. [Figure 9] This is a diagram showing an example of the state of presentation after the imaging operation of the abnormality detection result. [Figure 10] This is a diagram showing an example of the state of presentation after the imaging operation of the abnormality detection result. [Figure 11] This is a diagram showing an example of the state of guidance for re - imaging. [Figure 12] This is a diagram showing an example of data management of the re - imaging image. [Figure 13] This is a diagram showing an example of the appearance of the imaging system. [Figure 14] This is a block diagram showing the main configuration examples of each device. [Figure 15] This is a block diagram showing the main configuration examples of each device. [Figure 16] This is a block diagram showing the main configuration examples of each device. [Figure 17] This is a block diagram showing the main configuration examples of each device. [Figure 18] This is a flowchart explaining an example of the imaging process flow. [Figure 19] This is a flowchart explaining an example of the imaging process flow. [Figure 20] This is a flowchart following FIG. 19 and explaining an example of the imaging process flow. [Figure 21] This is a block diagram showing the main configuration example of a computer.

Embodiments for Carrying Out the Invention

[0011] Hereinafter, embodiments for implementing the present disclosure (hereinafter referred to as embodiments) will be described. The description will be given in the following order. 1. Documents etc. supporting technical content and technical terms 2. 3D modeling 3. Detection of abnormal images and utilization of detection results 4. First embodiment (imaging system) 5. Supplementary note

[0012] <1. Documents etc. supporting technical content and technical terms> The scope disclosed in this technology includes not only the content described in the embodiments, but also the content described in the following patent documents etc. that were known at the time of filing, and the content of other documents referred to in the following patent documents.

[0013] Patent Document 1: (described above) Patent Document 2: (described above)

[0014] That is, the content described in the above-mentioned patent documents and the content of other documents referred to in the above-mentioned patent documents also serve as a basis when determining the support requirements.

[0015] <2. 3D modeling> <Photogrammetry> Conventionally, for an object having a three-dimensional shape (also referred to as a 3D object in this specification), as a method for generating (reconstructing) a model of its three-dimensional shape, there has been a method called photogrammetry in which the 3D object is imaged from multiple directions and 3D data is generated based on the obtained plurality of imaging images. In this specification, generating a model of the three-dimensional shape of a 3D object is also referred to as 3D modeling.

[0016] Photogrammetry is a technique that uses the principle of triangulation to reconstruct a highly accurate three-dimensional model from multiple images taken from various viewpoints. In this specification, "accuracy" in relation to 3D data (3D models) may include not only the reproducibility (accuracy, detail, etc.) of the three-dimensional shape of the target 3D object, but also the reproducibility (accuracy, detail, etc.) of the texture applied to the surface of the 3D model. For example, as shown in Figure 1, a 3D object 10 is imaged from multiple viewpoints, from camera 11-1 to camera 11-5, to obtain multiple images. Then, using these images, a process called SfM (Structure from Motion) and a process called MVS (Multi-view Stereo) are performed, and further post-processing such as meshing and texturing is performed to generate 3D data 15.

[0017] In SfM, for example, corresponding points are searched between captured images, the camera's position and orientation are derived using epipolar constraints, and the position of each corresponding point in 3D space is determined by triangulation based on the camera's position and orientation. In this specification, these points in 3D space are also referred to as 3D points. In other words, the 3D point corresponding to each corresponding point is identified. Then, the entire set of 3D points identified in this way is optimized by bundle adjustment.

[0018] In MVS, for example, the 3D point cloud derived as described above is used to perform a more rigorous correspondence search and add more 3D points.

[0019] As described above, photogrammetry performs an overall optimization calculation called bundle adjustment to minimize errors, resulting in very high accuracy, but it is computationally intensive. Also, because it is based on geometric calculations rather than physical measurements, in principle, the higher the resolution of the images used, the more accurate the model that can be reconstructed.

[0020] <Retakes in photogrammetry imaging> The images used in such photogrammetry must be of high quality in order to more accurately reproduce the reality of the subject. However, since many images are required to generate 3D data, the imaging process requires capturing a large number of images in a short amount of time. Furthermore, in order to generate 3D data for 3D object 10, it is necessary to image a wider area of ​​the entire surface of 3D object 10, so it is necessary to repeat imaging while changing the viewpoint.

[0021] Therefore, it was difficult to take care to improve image quality in each imaging session. As a result, there was a risk that excessive blurring or blurring would occur frequently in the captured images due to camera performance limitations or photographer errors. In this specification, captured images containing such excessive blurring or blurring will also be referred to as "abnormal images." In other words, in photogrammetry imaging, there was a risk that the frequency of captured images becoming abnormal images would be high.

[0022] To generate more accurate 3D data, it may be necessary to re-capture any abnormal images. However, in the case of photogrammetry, as mentioned above, it is necessary to repeat the imaging process many times while changing the imaging viewpoint (location and orientation of the image), making it difficult to check the quality of the image obtained with each imaging.

[0023] Incidentally, Patent Documents 1 and 2 disclose methods for detecting and notifying of such abnormal images. For example, Patent Document 1 discloses a system in which a camera is mounted on a pan / tilt head and imaging is performed while the pan / tilt head is moved, in which, after the shooting is completed, images that failed to be captured are detected, and the pan / tilt head is controlled to retake those images. Patent Document 2 also discloses a system in which a multicopter is flown to photograph a field and generate 3D data of the field, in which, after the shooting of the field is completed, the data of the captured images is judged, and the system notifies of parts where invalid data exists or data is missing, and controls the flight path for reshooting.

[0024] However, the methods described in these patent documents were methods for detecting abnormal images after the imaging process was completed. Furthermore, in these methods, the imaging conditions applied to each image were predetermined, and it was relatively easy to retake the image by setting the imaging conditions similarly. Here, "imaging conditions" refers to the conditions that describe how imaging is performed, and can include any conditions related to imaging. For example, these "imaging conditions" may include imaging parameters and imaging viewpoints (location and orientation of imaging). Therefore, in the case of imaging work for photogrammetry as described above, these methods could make it difficult to retake images of detected abnormal images.

[0025] For example, in photogrammetry imaging, numerous images are taken. Furthermore, the imaging conditions are determined for each image. Therefore, identifying the imaging conditions for images that yielded abnormal images requires a complicated process, and it may be difficult to accurately reproduce those imaging conditions. In other words, it may be difficult to perform correct retakes. Moreover, even if retakes are performed successfully, the order of the captured images may be disrupted, making 3D model synthesis difficult for 3D model generation software that requires correspondence between adjacent images.

[0026] <3. Detection of abnormal images and utilization of detection results> <Method 1> Therefore, as shown in the top row of the table in Figure 2, in photogrammetry imaging, anomaly detection is performed immediately (in real time) on the captured image, and the captured image and the anomaly detection result are associated and stored (Method 1). Meta information such as the location, orientation, and shooting parameters of the image are linked to the image, and any anomalies in the captured image are detected. By processing the detection of anomaly images and the normal shooting assist function in parallel, real-time (immediacy) is maintained.

[0027] For example, the information processing device may include an anomaly detection unit that immediately performs anomaly detection on an image captured for generating three-dimensional shape information representing the three-dimensional shape of a 3D object, and a storage unit that stores the image captured and the results of the anomaly detection in association with each other.

[0028] For example, the information processing method executed by the information processing device includes, immediately upon generating an image for which three-dimensional shape information representing the three-dimensional shape of a 3D object is generated, performing anomaly detection on the image, and storing the image and the result of the anomaly detection in association with each other.

[0029] For example, a program can be configured to cause a computer to perform a process that includes immediately detecting anomalies in an image captured for generating three-dimensional shape information representing the three-dimensional shape of a 3D object, and storing the image and the results of the anomaly detection in association with each other.

[0030] As shown in Figure 3A, anomaly detection processing is immediately performed on the captured image generated by the imaging process, and the captured image and the anomaly detection result are stored in association with each other. In other words, as shown in Figure 3B, the image data and its anomaly detection result are managed in correspondence. In the example in Figure 3B, the detection result for "03.jpg" is "Blurred," indicating that it is an anomaly image.

[0031] This approach allows users to easily retake images with the exact same conditions if an abnormal image is captured due to limitations in the camera's autofocus performance or a photographer's error, without having to remember the location or camera settings. Furthermore, since abnormal images can be automatically detected without the photographer having to check each image, the inclusion of abnormal images can be suppressed, thus preventing a decrease in the quality of 3D data generated by photogrammetry. It also helps to minimize a decrease in the efficiency of the shooting process. Additionally, by performing anomaly detection immediately during image capture, anomaly detection can be performed in parallel with the shooting process, reducing the increase in waiting time due to image transfer for anomaly detection, for example. In short, users can more easily retake images with abnormalities during photogrammetry.

[0032] The definition of an abnormal image (what constitutes an abnormal image) can be anything. In other words, any captured image may be detected as an abnormal image in anomaly detection. For example, an excessively blurred captured image, that is, an image that is blurred above a predetermined standard, may be determined to be an abnormal image. Similarly, an excessively smudged captured image, that is smudged above a predetermined standard, may be determined to be an abnormal image. Furthermore, an image in which both blurring and smudgement are excessive may be determined to be an abnormal image. For example, in an information processing device, an anomaly detection unit may detect excessive blurring and smudgement in an captured image as abnormal.

[0033] Furthermore, any method can be used to detect abnormal images. For example, when detecting excessive blur or blur as an abnormal image, as described above, the following algorithm may be applied.

[0034] First, the second derivative of the image is calculated using a Laplacian filter to emphasize abrupt changes in edges and textures. In areas where blurring or motion blur occurs, the edge intensity weakens, resulting in a lower variance score after applying the filter. Therefore, the variance score is calculated, and the lower the variance, the more blurred or motion blurred the image is judged to be.

[0035] The generation of a Laplacian image (edge-enhanced image) involves convolving a 3x3 Laplacian kernel onto the image. Specifically, the kernel center is aligned with each pixel, and the sum of the products of the kernel value and the corresponding pixel value in the image is calculated. By performing this operation across the entire image, a Laplacian image is generated.

[0036] The variance is calculated using the following equation (1). In equation (1), σ^2 is the variance, N is the number of pixels, x_i is the value of each pixel, and μ is the mean of the pixel values. In this specification, "^" indicates an exponent. That is, "A^B" indicates that the base is A and the exponent is B. Also, "_" indicates a subscript. That is, "A_B" indicates that B is a subscript of A.

[0037]

number

[0038] The pixel values ​​in a Laplacian image represent the edge strength at each point in the original image. The sharper the edges, the larger the absolute value of the pixel. Therefore, the overall sharpness of an image can be quantitatively evaluated by calculating the variance of the pixel values ​​in a Laplacian image.

[0039] Furthermore, in anomaly detection, a determination may be made as to whether or not the image is abnormal. In addition, a determination of the degree of abnormality may also be made. For example, based on anomaly detection, the captured image may be classified into three stages: always normal, abnormal depending on the required quality, and always abnormal. The degree of abnormality may also be indicated numerically. In addition, the type of abnormality (e.g., blur, blur, etc.) may be identified.

[0040] Furthermore, anomaly detection may be performed on the entire captured image, or on a portion of the captured image, such as the central part.

[0041] Furthermore, when storing captured images, not only the results of anomaly detection but also information related to the imaging may be stored in association with the captured image. For example, in an information processing device, the storage unit may also store imaging-related information related to the imaging that generated the captured image, in association with the captured image.

[0042] This imaging-related information can be any information related to imaging. For example, this imaging-related information may include information about the camera's position and orientation during imaging (i.e., information about the imaging viewpoint). It may also include imaging parameters applied to the imaging. For example, in the case of C in Figure 3, the imaging position (coordinates) is associated with the image as imaging-related information. By associating such information with the captured image, for example, when re-imaging an abnormal image, the imaging conditions can be reproduced more easily and accurately based on that information.

[0043] Furthermore, this technology may be applied to devices having an imaging function, i.e., imaging devices. For example, an information processing device may further include an imaging unit that images a subject and generates an image. The anomaly detection unit may then immediately perform anomaly detection on the image captured by the imaging unit.

[0044] <Method 1-1> When Method 1 described above is applied, the anomaly detection results may be presented immediately (in real time), as shown in the second row from the top of the table in Figure 2 (Method 1-1). For example, as shown in Figure 4A, the detection result presentation process may be executed immediately in response to the anomaly detection process.

[0045] For example, the information processing device may further include a detection result presentation processing unit that immediately presents the result of an anomaly detection. In this way, the user (imager) performing the imaging operation can immediately grasp that an abnormal image has occurred (a low-quality image has been generated) during the imaging operation. Therefore, the user can immediately retake the image (re-imaging). In other words, the user can retake the image as part of the imaging workflow. Furthermore, the user can perform the re-imaging operation while having a more accurate understanding of information regarding imaging conditions, such as imaging position and orientation (i.e., imaging viewpoint) and imaging parameters. Therefore, the user can perform the re-imaging operation more easily and accurately. In short, the user can more easily retake abnormal images in photogrammetry imaging.

[0046] Furthermore, the information processing device may also include a re-imaging guidance request receiving unit that receives requests for guidance on re-imaging under the same imaging conditions as the image in which an anomaly was detected. In this way, the user can request guidance on re-imaging based on the presentation of the anomaly detection result.

[0047] The method for immediately displaying the results of anomaly detection can be any method. For example, when an abnormal image is detected, a pop-up screen 113 indicating that the anomaly has been detected may be displayed on the display unit of the imaging device 111 (e.g., a smartphone) superimposed on the captured image 112, as shown in Figure 4B. When the user selects the pop-up screen 113, the detected abnormal image 114 may be displayed, as shown in Figure 4C. The method for selecting the pop-up screen 113 can be any method. For example, the display unit may be equipped with a touch panel, and the user may use the touch panel to select the pop-up screen 113. Alternatively, if the user does not select the pop-up screen 113 within a predetermined time, the display of the pop-up screen 113 may be terminated. By doing so, it is possible to prevent the immediate display of anomaly detection results from interfering with the imaging work.

[0048] Furthermore, if the user selects an abnormal image 114, a detailed confirmation screen 115 for that abnormal image may be displayed, as shown in Figure 5A. In the example of Figure 5A, this detailed confirmation screen 115 displays the project (Project: SCMM_Port_Planter_3), image ID (Photo ID: 0208 / 1223), image file name (DSC003112), image format (Format: JPEG + RAW), capture date and time (Data: 2024 / 3 / 28 12:38), aperture (IRIS: F8.0), shutter speed (SS: 1 / 200), ISO sensitivity (ISO: 100), focal length (Focal Length: 20mm), image size (Size: 61MP 6240x4160 20MB), camera equipment information (Camera Info: ILCE-7R V), lens information (Lens: 35mm F1.4 GM), etc. Furthermore, the information displayed on this detailed confirmation screen 115 may be any information related to the abnormal image 114.

[0049] Furthermore, if the user presses the re-imaging guidance button 116, the request may be accepted and guidance for re-imaging may be provided.

[0050] Furthermore, as shown in the example in Figure 5B, when an abnormal image is detected, the detected abnormal image 114 may be displayed superimposed on the captured image 112 along with the pop-up screen 113. In this case, if the user selects either the pop-up screen 113 or the abnormal image 114, the system may transition to a detailed confirmation screen 115 as shown in Figure 5A. Also, in the case of Figure 5B, if the user does not select either the pop-up screen 113 or the abnormal image 114 within a predetermined time, the display of the pop-up screen 113 and the abnormal image 114 may be terminated. By doing so, it is possible to prevent the immediate presentation of abnormality detection results from interfering with the imaging work.

[0051] <Method 1-2> When Method 1 described above is applied, the anomaly detection results may be presented after or during the imaging process, as shown in the third row from the top of the table in Figure 2 (Method 1-2). For example, as shown in Figure 6A, the detection result presentation process may be performed on the captured images (which may include anomaly images) stored in the memory unit.

[0052] For example, the information processing device may further include a detection result presentation processing unit that presents the results of anomaly detection stored in the storage unit after the completion or interruption of the imaging operation that generates the captured image.

[0053] For example, a smartphone application could display abnormal images as a list, or show them in detail, and allow users to interact with that display. Generally, in a system configuration that links a smartphone with an ILC (Interchangeable Lens Camera), operations and confirmations of captured images must be performed on the ILC screen. However, by giving the smartphone such functionality, all operations can be completed on the smartphone, making data management easier and mitigating the reduction in work efficiency caused by such operations on captured images.

[0054] In other words, users can more easily retake abnormal images during photogrammetry imaging.

[0055] Furthermore, the information processing device may also include a re-imaging guidance request receiving unit that receives requests for guidance on re-imaging under the same imaging conditions as the image in which an anomaly was detected. In this way, the user can request guidance on re-imaging based on the presentation of the anomaly detection result.

[0056] <Method 1-2-1> Furthermore, any method may be used to immediately present the results of anomaly detection. For example, when Method 1-2 described above is applied, the anomaly detection results may be superimposed onto the real space, as shown in the fourth row from the top of the table in Figure 2 (Method 1-2-1).

[0057] For example, in an information processing device, the detection result presentation processing unit may superimpose the results of anomaly detection onto the real space. Alternatively, the detection result presentation processing unit may superimpose the results of the anomaly detection onto the real space as seen from the current camera's viewpoint.

[0058] For example, when an abnormal image is detected, the captured image (including the abnormal image) may be superimposed on the captured image 122 displayed on the display unit of the imaging device 121 (e.g., a smartphone), as shown in Figure 6B. This captured image 122 is an image of real space. In other words, this captured image 122 can be said to be an image of real space as seen from the current camera's viewpoint. In the example of Figure 6B, the abnormality detection results for captured images 123-1, 123-2, and 123-3 are superimposed on this captured image 122. Captured images 123-1 and 123-3 are shown with solid line frames, indicating that they were normal images (no abnormalities were detected). Captured image 123-2 is shown with a dotted line frame, indicating that it was an abnormal image (an abnormality was detected). In other words, the type of dotted line frame indicates the abnormality detection result. That is, the abnormality detection result is presented superimposed on the real space as seen from the current camera's viewpoint.

[0059] In this example, captured images 123-1, 123-2, and 123-3 are positioned in the acquired image 122 to correspond to their respective imaging positions and orientations (i.e., imaging viewpoints). In other words, the positions and orientations of captured images 123-1, 123-2, and 123-3 indicate the imaging viewpoints for each image. Therefore, the user can easily identify which part of the image acquisition failed (where an abnormal image occurred). This means the user can easily determine which part needs to be re-imaging. Consequently, the user can more easily re-image abnormal images in photogrammetry.

[0060] Furthermore, the method of presenting the position and orientation of the captured image (the imaging viewpoint used to obtain that image) can be any method and is not limited to the example in Figure 6B (rectangular frame). For example, it may be shown as a hexagonal frame, as in the example in Figure 6C. Also, the captured image may or may not be displayed within the frame.

[0061] Furthermore, the method for presenting the anomaly detection result (for example, identifying whether an image is abnormal or normal) can be any method and is not limited to the examples in Figure 6B and Figure 6C (i.e., identification by the type of frame line). For example, the anomaly detection result may be indicated by the color of the frame line. For example, the color of the frame line may be different for normal images and abnormal images. Alternatively, as shown in Figure 7A, the anomaly detection result may be indicated by the color (fill) inside the frame. For example, the inside of the frame of an abnormal image may be filled with a predetermined color. Alternatively, as in the example in Figure 7B, the anomaly detection result may be indicated by the thickness of the frame line. For example, the frame line of an abnormal image may be thicker than that of a normal image. Alternatively, the anomaly detection result may be indicated by the shape of the frame. For example, the frame shape may be different for normal images and abnormal images. Alternatively, as in the example in Figure 8A, an image indicating the anomaly detection result may be attached. In this example, the captured image 123-2 is attached with a "?" image, indicating that this image is an abnormal image (that an anomaly was detected).

[0062] Furthermore, as an anomaly detection result, in addition to identifying whether an image is normal or abnormal, the degree of abnormality may also be indicated, as in example B in Figure 8. In this example, the degree of abnormality of captured images 123-2 and 123-4 is indicated by patterns or characters. By using such a display, users can more easily grasp the degree of abnormality of each captured image. The type of abnormality (e.g., blur, overexposure, underexposure, etc.) may also be indicated.

[0063] As described above, in the state where the anomaly detection result (anomaly image) is superimposed on the captured image 122, if the user selects the presented anomaly image, the system may transition to a screen to confirm its detailed information.

[0064] The method for selecting the abnormal image can be any method. For example, the display unit may be equipped with a touch panel, and the user may use the touch panel to select the abnormal image. For example, the user may touch the frame of the abnormal image once, and then be taken to a screen to confirm its detailed information. Alternatively, the frame of the abnormal image may blink when the user touches it once, and then touching it again may take the user to a screen to confirm its detailed information. Furthermore, a separate button for confirming details may be provided, and the user may take the user to a screen to confirm the detailed information by pressing this button.

[0065] Furthermore, as shown in Figure 9A, for example, if a user brings the imaging device 121 closer to the location of the abnormal image, image 123-2, in real space, as indicated by arrow 131, the frame of image 123-2 may blink. In this state, if the user touches the area within the frame of image 123-2 once, the system may transition to a screen for confirming its detailed information.

[0066] As described above, when an abnormal image is selected, a detailed confirmation screen 132 may be displayed, as shown in the example in Figure 9B. The information displayed on this detailed confirmation screen 132 can be any information, as in the case of the detailed confirmation screen 115. Furthermore, if the user presses the re-imaging guidance button 133, the request may be accepted and guidance for re-imaging may be provided.

[0067] <Method 1-2-2> For example, when Method 1-2 described above is applied, the anomaly detection results may be superimposed on the virtual space and presented as shown in the fifth row from the top of the table in Figure 2 (Method 1-2-2).

[0068] For example, in an information processing device, the detection result presentation processing unit may superimpose the results of anomaly detection onto a virtual space and present them.

[0069] In a virtual space, the viewpoint from which the anomaly detection results are superimposed becomes a virtual viewpoint. This virtual viewpoint can be freely set. In other words, this virtual viewpoint can be any viewpoint. For example, it could be a bird's-eye view, a position that provides an overview. For example, in an information processing device, the detection result presentation processing unit may superimpose the anomaly detection results onto a virtual space viewed from a predetermined bird's-eye viewpoint and present them.

[0070] An example of how the anomaly detection results are presented in this case is shown in Figure 10A. In the example in Figure 10A, an image of a virtual space (a so-called bird's-eye view image) 142 viewed from a predetermined overhead viewpoint is displayed on the display unit of the imaging device 141 (for example, a smartphone). The 3D objects in this virtual space (i.e., the objects displayed in image 142) correspond to 3D data generated using captured images, etc., by photogrammetry or the like. The captured image 143 is then superimposed on the image 142 of the virtual space. This captured image 143 is presented in a way that indicates the anomaly detection results, similar to the example explained with reference to Figures 6 to 9. For example, the captured image 143 of a normal image is shown with a solid line frame, while the captured image 143-1 of an abnormal image is shown with a dotted line frame. In other words, the results of the anomaly detection are presented superimposed on the virtual space viewed from a predetermined overhead viewpoint.

[0071] The method for transitioning to the detailed confirmation screen is the same as the example explained with reference to Figures 6 to 9.

[0072] By presenting the anomaly detection results in this way, users can easily understand which part of the image acquisition failed (where an abnormal image occurred). In other words, users can easily understand which part needs to be re-captured (re-imaging). Therefore, users can more easily re-capture abnormal images in photogrammetry.

[0073] Alternatively, for example, a virtual camera may be set up in a virtual space, and the viewpoint of that virtual camera may be applied. For example, in an information processing device, the detection result presentation processing unit may superimpose the anomaly detection result onto the virtual space as seen from the viewpoint of the virtual camera. In this case, the method of presenting the anomaly detection result is the same as the example described with reference to Figures 6 to 9. Therefore, even with this method, the user can easily understand which part of the image acquisition failed (where an abnormal image occurred). In other words, the user can easily understand which part needs to be re-captured (re-imaging). Therefore, the user can more easily re-capture abnormal images in photogrammetry.

[0074] However, in the case of a virtual camera viewpoint, control of that viewpoint (such as movement) may be performed by camera operation or by screen operation (such as touch panel operation).

[0075] <Method 1-2-3> For example, when Method 1-2 described above is applied, the anomaly detection results may be presented in the list of captured images, as shown in the sixth row from the top of the table in Figure 2 (Method 1-2-3).

[0076] For example, in an information processing device, a detection result presentation processing unit may present the results of anomaly detection in a list of captured images. This makes it easier for the user to retake abnormal images during photogrammetry imaging.

[0077] For example, as shown in Figure 10B, a list screen 152 of the captured images 153 may be displayed on the display unit of the imaging device 151 (e.g., a smartphone), and the anomaly detection results may be presented on the list screen 152. In the example of Figure 10B, captured images 153-1 and 153-2, which are abnormal images (anomalies have been detected), are marked with a flag image indicating that they are abnormal images (an anomaly has been detected). This flag image allows the user to identify the abnormal images. Of course, the method for distinguishing between normal and abnormal images can be any method, and is the same as the example described with reference to Figures 6 to 9. In addition, the detected degree of anomaly and the type of anomaly may also be presented.

[0078] <Method 1-3> For example, when Method 1 described above is applied, the system may guide the user to re-imaging based on the anomaly detection result, as shown in the seventh row from the top of the table in Figure 2 (Method 1-3). The location and orientation for re-imaging may be displayed as a UI (User Interface) to provide visual guidance. In this way, the user does not need to decipher the anomaly image or other re-imaging information and search for the location and orientation for re-imaging, and can easily and correctly re-imaging simply by following the guidance. For example, the location where the anomaly image was captured may be projected and visualized in real space. In this way, the anomaly image and the surrounding space can be viewed simultaneously in real space, providing more intuitive assistance in deciding whether to re-image.

[0079] For example, the information processing device may further include a re-imaging guidance processing unit that guides the user to re-image an image under the same imaging conditions as the image in which an anomaly was detected. For example, the re-imaging guidance processing unit may superimpose the area to be re-imaged onto the real space. For example, the re-imaging guidance processing unit may superimpose the area to be re-imaged onto the real space as seen from the current camera's viewpoint.

[0080] For example, as described above, if a re-imaging guidance is requested, such as by pressing the re-imaging guidance button, the re-imaging guidance may be started in response to that request. Any method can be used to guide the re-imaging. For example, a guidance screen like the one shown in Figure 11A may be displayed. In this example, the captured image 162 is displayed on the display unit of the imaging device 161 (e.g., a smartphone), and the re-imaging position 164 is superimposed on the captured image 162. This re-imaging position 164 indicates the position and orientation (i.e., the imaging viewpoint) of the image in which the abnormal image occurred. In other words, it corresponds to the position and orientation of the captured image in which the abnormality was detected, as described with reference to Figures 6 to 9, etc.

[0081] This captured image 162 is an image of real space. In other words, this captured image 162 can be said to be an image of real space as seen from the current camera's viewpoint. That is, the area being re-captured is superimposed on the real space as seen from the current camera's viewpoint.

[0082] The user simply moves the imaging device 161 to match the re-imaging position 164 and performs re-imaging at that position and orientation. This guidance allows the user to perform re-imaging (retakes) more easily and accurately. In other words, the user can more easily re-image abnormal images in photogrammetry.

[0083] Furthermore, when the user aligns the imaging device 161 with the re-imaging position 164, a notification indicating that the position has been corrected may be provided. The method of this notification may be any. For example, the display of the re-imaging position 164 (e.g., color or pattern) may change. Also, the color and brightness of the entire screen may change.

[0084] Furthermore, re-imaging may be performed by the user inputting an instruction to perform re-imaging, such as by pressing the shutter button. For example, the information processing device may further include a re-imaging instruction receiving unit that receives an instruction to perform re-imaging.

[0085] Furthermore, during this re-imaging, the imaging parameters applied in the previous imaging (the imaging in which the abnormal image occurred) may be automatically applied. Alternatively, the imaging device 161 may be configured to perform re-imaging simply by the user moving it to the re-imaging position 164 (without the user pressing the shutter button). For example, re-imaging may be performed when the user fixes the imaging device 161 at the re-imaging position 164 for a predetermined time. This makes the re-imaging process easier for the user.

[0086] In the example shown in Figure 11A, the re-imaging location 164 is indicated by a hexagon, but the presentation of this re-imaging location is not limited to this example and may be done in any way. For example, the re-imaging location may be indicated by a square frame or by a predetermined image. Also, in the example shown in Figure 11A, the re-imaging location 164 is presented together with the other normal images, image 163-1 and image 163-3. In other words, in the presentation of the abnormality detection result in the example explained with reference to Figures 6 to 9, the re-imaging location 164 is shown at the location of the image determined to be an abnormal image. The method of presenting the re-imaging location is not limited to this example, and for example, as shown in Figure 11B, only the re-imaging location 164 may be presented without presenting the normal images, image 163-1 and image 163-3.

[0087] <Method 1-4> For example, when Method 1 described above is applied, the re-imaging of the abnormal image may be replaced with the original image, as shown in the bottom row of the table in Figure 2 (Method 1-4). In other words, as described above, when re-imaging is performed, the image obtained from that re-imaging may be stored in place of the abnormal image obtained from the previous imaging at the same location. In other words, the re-imaging image and its metadata may replace the original image, thereby maintaining the order and continuity of the image group.

[0088] For example, in an information processing device, the storage unit may store a re-captured image obtained by re-capturing the image in which the anomaly was detected under the same imaging conditions as the original image in which the anomaly was detected, replacing the original image in which the anomaly was detected.

[0089] Generally, re-captured images are added to the image set as the most recent. For example, suppose that photogrammetry captures 15 captured images 171 as shown in Figure 12A. In Figure 12, the numbers within the captured images 171 indicate the capture order. Suppose that anomalies are detected in the 7th and 14th captured images 171 during anomaly detection processing. Generally, if these images are recaptured, the re-captured images 172 and 173 are managed as the 16th and 17th images, as shown in Figure 12B. However, the re-captured images 172 and 173 are recaptures of the 7th and 14th captured images 171. Therefore, with the management method in Figure 12B, the capture order will be different from that in Figure 12A. As a result, the correspondence between adjacent images will be different from that of the previous image (anomalous image). Therefore, it was necessary to manually adjust the correspondence in the subsequent 3D model synthesis flow.

[0090] Therefore, the images are replaced and stored as described above. In other words, as shown in Figure 12C, the re-captured images 172 and 173 are replaced with the 7th and 14th captured images 171 and stored. By doing this, the imaging order can be maintained even after re-capture, thereby improving the overall workflow efficiency of 3D model synthesis through autonomous response. In short, users can more easily re-capture abnormal images in photogrammetry imaging.

[0091] For example, when storing a re-captured image obtained through re-capture, the original image (abnormal image) obtained from the previous capture at the same location may be discarded, and its filename may be applied to the re-captured image. For example, in an information processing device, the storage unit may apply the filename of the captured image in which an abnormality was detected as the filename of the re-captured image.

[0092] It is common practice to assign a number to the file name that indicates the imaging order. In such cases, by reusing the file name, re-imaging images can be stored in the same imaging order as the original images. In other words, abnormal images can be replaced with re-imaging images while maintaining the imaging order.

[0093] Furthermore, when storing the re-captured image obtained through re-capture, the original image (abnormal image) obtained from the previous capture at the same location may be discarded, and its metadata may be applied to the re-captured image. For example, in an information processing device, the storage unit may apply the metadata of the captured image in which an abnormality was detected as the metadata of the re-captured image.

[0094] If metadata includes information such as the imaging order and imaging parameters, this information can be carried over, allowing abnormal images to be replaced with re-imaged images while maintaining the imaging order.

[0095] <Application Examples> In the above explanation, for example, the presentation of anomaly detection results and guidance for re-imaging were described as being performed by image display, but these can be implemented by any method and are not limited to image display. For example, anomaly detection results and guidance for re-imaging may be performed using output such as sound or vibration. Furthermore, multiple methods may be combined, such as combining image display with sound output or vibration output.

[0096] Alternatively, after the anomaly detection process, the system may present the anomaly detection results without storing the captured image or the anomaly detection results.

[0097] Furthermore, anomaly detection processing may be performed (non-immediately) after the captured image has been stored. In this case as well, by displaying the anomaly detection results and providing guidance for re-imaging, as in the example above, users can more easily re-capture abnormal images during photogrammetry.

[0098] <Scope of application of the explanation> In this specification, explanations given for higher-level methods also apply to lower-level methods belonging to that method, provided that they do not create a contradiction. For example, if it is stated that "Method 1 may be applied," it means that any of Methods 1-1, 1-2, 1-3, or 1-4 may be applied. Of course, even lower-level methods (e.g., Methods 1-2-1, 1-2-2, 1-2-3, etc.) may also be applied.

[0099] <Combinations> Furthermore, each of the methods described above may be applied in combination with any other method, provided that no contradiction arises. Three or more methods may be applied in combination. In addition, the combinatorial methods may include not only those shown in the table in Figure 2, but all elements described herein. Moreover, each of the methods described above may be applied in combination with other methods not described above. <4. First Embodiment> <Imaging System> This technology can be applied to any configuration (device, system, etc.). For example, it can be applied to an imaging system consisting of an imaging communication device 301 and an imaging device 302, as shown in Figure 13.

[0100] The imaging communication device 301 is a device having imaging, communication, and information processing functions. The imaging communication device 301 may be configured as, for example, a smartphone. This imaging communication device 301 has a depth sensor 311, an imaging unit 312, and an IMU (Inertial Measurement Unit) (not shown). The imaging device 302 is a device having imaging functions. The imaging device 302 may be configured as, for example, an ILC with interchangeable optical systems such as lenses. The imaging communication device 301 and the imaging device 302 are connected to each other so as to be able to communicate with one another, and can exchange information through this communication. The imaging communication device 301 can also be installed at a predetermined position on the imaging device 302. In other words, the imaging communication device 301 is detachable from the imaging device 302.

[0101] Figure 14 is a block diagram showing an example of the main configuration of such an imaging communication device 301 and imaging device 302. Note that Figure 14 shows the main components such as processing units and data flows, and is not necessarily exhaustive. In other words, the imaging communication device 301 and imaging device 302 may have devices and processing units that are not shown as blocks in Figure 14. Also, there may be data flows and processes that are not shown as arrows or other symbols in Figure 14.

[0102] As shown in Figure 14, in this case, the imaging communication device 301 includes a depth sensor 311, an imaging unit 312, an IMU 313, a 3D attitude derivation unit 314, an imaging control unit 315, an anomaly detection unit 316, a data management unit 317, a storage unit 318, a detection result presentation processing unit 319, a re-imaging guidance unit 320, a display unit 321, and an input unit 322. The imaging device 302 also includes an imaging unit 331.

[0103] The depth sensor 311 has a Lidar sensor (dToF module), etc., detects the depth to the subject, and supplies it to the 3D attitude derivation unit 314. The imaging unit 312 has an image sensor, captures an image of the subject, generates an image, and supplies it to the 3D attitude derivation unit 314. The IMU 313 detects the inertial information (acceleration and angular velocity) of the imaging device and supplies it to the 3D attitude derivation unit 314.

[0104] The 3D attitude derivation unit 314 derives the position and attitude of the imaging communication device 301 and the imaging device 302 based on the information supplied from them. The 3D attitude derivation unit 314 supplies the derived position and attitude information to processing units such as the imaging control unit 315, the detection result presentation processing unit 319, and the re-imaging guidance unit 320.

[0105] The imaging control unit 315 controls the operation of the imaging unit 331 of the imaging device 302. For example, the imaging control unit 315 may control the imaging unit 331 to image a subject and generate an image. The imaging control unit 315 may also control the imaging unit 331 to perform re-imaging (re-taking of abnormal images). The imaging control unit 315 may also control the imaging unit 331 based on information indicating the position and orientation of the imaging communication device 301 and the imaging device 302 supplied by the 3D orientation derivation unit 314. Furthermore, the imaging control unit 315 may also control the imaging unit 331 based on information supplied by the re-imaging guidance unit 320 (for example, a re-imaging instruction input via the input unit 322 during re-imaging guidance).

[0106] The anomaly detection unit 316 performs processing related to anomaly detection of the captured image. For example, the anomaly detection unit 316 may perform anomaly detection on the captured image immediately upon its generation to generate three-dimensional shape information representing the three-dimensional shape of a 3D object. For example, the anomaly detection unit 316 may acquire the captured image generated by the imaging unit 331 and perform anomaly detection on the captured image immediately upon imaging by the imaging unit 331. For example, the anomaly detection unit 316 may detect excessive blurring and shaking in the captured image as anomalies. The anomaly detection unit 316 may supply the captured image and the results of its anomaly detection to the data management unit 317 and the detection result presentation processing unit 319.

[0107] The data management unit 317 manages the information to be stored in the storage unit 318. For example, the data management unit 317 may acquire captured images and anomaly detection results supplied from the anomaly detection unit 316 and store them in the storage unit 318. In this case, the data management unit 317 may associate the anomaly detection results of the captured image with the captured image and store them in the storage unit 318. Alternatively, the data management unit 317 may associate imaging-related information concerning the imaging in which the captured image was generated with the captured image and store it in the storage unit 318.

[0108] The storage unit 318 stores information managed by the data management unit 317. For example, the storage unit 318 may store the captured image and the anomaly detection result in association with each other. Furthermore, the storage unit 318 may also store imaging-related information related to the imaging that generated the captured image, in association with that captured image. The imaging-related information may include information about the camera's position and orientation during imaging. The imaging-related information may also include imaging parameters applied to the imaging. The storage unit 318 may supply this stored information to the detection result presentation processing unit 319 and the re-imaging guidance unit 320. The storage unit 318 may replace the captured image in which the anomaly was detected with a re-imaging image obtained by re-imaging under the same imaging conditions as the captured image in which the anomaly was detected, and store it. In this case, for example, the storage unit 318 may use the file name of the captured image in which the anomaly was detected as the file name of the re-imaging image. The storage unit 318 may also use the metadata of the captured image in which the anomaly was detected as the metadata of the re-imaging image. In other words, the data management unit 317 may control the system in this way to store the re-captured image in the storage unit 318.

[0109] The detection result presentation processing unit 319 performs processing related to the presentation of detection results. For example, the detection result presentation processing unit 319 may supply the detection results read from the storage unit 318 to the display unit 321 for presentation. For example, the detection result presentation processing unit 319 may immediately present the results of an anomaly detection to the display unit 321 in response to an anomaly detection. Alternatively, the detection result presentation processing unit 319 may read the results of an anomaly detection stored in the storage unit 318 and present them to the display unit 321 after the completion or interruption of the imaging operation that generates the captured image. For example, the detection result presentation processing unit 319 may superimpose the results of the anomaly detection onto the real space and present them to the display unit 321. Alternatively, the detection result presentation processing unit 319 may superimpose the results of the anomaly detection onto the real space as seen from the current camera viewpoint and present them to the display unit 321. Alternatively, the detection result presentation processing unit 319 may superimpose the results of the anomaly detection onto a virtual space and present them to the display unit 321. Furthermore, the detection result presentation processing unit 319 may superimpose the results of the anomaly detection onto a virtual space viewed from a predetermined overhead viewpoint and display them on the display unit 321. Alternatively, the detection result presentation processing unit 319 may superimpose the results of the anomaly detection onto a virtual space viewed from the viewpoint of a virtual camera and display them on the display unit 321. Furthermore, the detection result presentation processing unit 319 may display the results of the anomaly detection on the display unit 321 in a list of captured images. Additionally, the detection result presentation processing unit 319 may receive a request for re-imaging guidance via the input unit 322, and control the re-imaging guidance unit 320 to execute the re-imaging guidance in response to the request.

[0110] The re-imaging guidance unit 320 performs processing related to guidance for re-imaging. For example, the re-imaging guidance unit 320 may supply guidance for re-imaging (re-taking of abnormal images) to the display unit 321 based on information read from the storage unit 318 and have it displayed. In other words, the re-imaging guidance unit 320 can be said to be a re-imaging guidance processing unit that guides the user to re-image the image under the same imaging conditions as the image in which an abnormality was detected due to abnormality detection. For example, the re-imaging guidance unit 320 may superimpose the area to be re-imaged onto the real space and have it displayed on the display unit 321. Alternatively, the re-imaging guidance unit 320 may superimpose the area to be re-imaging onto the real space as seen from the current camera's viewpoint and have it displayed on the display unit 321. Furthermore, the re-imaging guidance unit 320 may receive a re-imaging instruction input via the input unit 322, supply that instruction to the imaging control unit 315, and have it perform re-imaging.

[0111] The display unit 321 has a display device and performs the display of information supplied from the detection result presentation processing unit 319, the re-imaging guidance unit 320, etc. For example, the display unit 321 may immediately present the result of an anomaly detection. Alternatively, the display unit 321 may present the result of the anomaly detection read from the storage unit 318 after the completion or interruption of the imaging operation that generates the captured image. For example, the display unit 321 may superimpose the result of the anomaly detection onto the real space. Alternatively, the display unit 321 may superimpose the result of the anomaly detection onto the real space as seen from the current camera viewpoint. Alternatively, the display unit 321 may superimpose the result of the anomaly detection onto a virtual space. Alternatively, the display unit 321 may superimpose the result of the anomaly detection onto a virtual space as seen from a predetermined overhead viewpoint. Alternatively, the display unit 321 may superimpose the result of the anomaly detection onto a virtual space as seen from the viewpoint of a virtual camera. Furthermore, the display unit 321 may present the results of the anomaly detection in a list of captured images. The input unit 322 has an input device and accepts instructions and information input from, for example, a user. For example, the input unit 322 may accept a request for guidance on re-imaging under the same imaging conditions as the captured image in which an anomaly was detected. Therefore, the input unit 322 can also be called a re-imaging guidance request receiving unit. The input unit 322 may also accept instructions to perform re-imaging. Therefore, the input unit 322 can also be called a re-imaging instruction receiving unit. The input unit 322 supplies the received instructions and information to the detection result presentation processing unit 319, the re-imaging guidance unit 320, etc.

[0112] The imaging unit 331 has an image sensor and uses the image sensor to capture an image of the subject and generate an image. The imaging unit 331 supplies the generated image to the anomaly detection unit 316.

[0113] The above describes an example consisting of an imaging communication device 301 and an imaging device 302. However, this technology can also be realized by an imaging device 302 having sensor functionality, for example, by attaching an external module. A main configuration example of the imaging device 302 in that case is shown in Figure 15. As shown in Figure 15, the imaging device 302 in this case has the same configuration as in Figure 14 (the imaging communication device 301 and imaging device 302 in the example of Figure 14). In the example of Figure 15, the depth sensor 311 and IMU 313 are configured as an external module 341.

[0114] Of course, this sensor function may be built into the imaging device 302. A typical example of the main configuration of the imaging device 302 in that case is shown in Figure 16. As shown in Figure 16, the imaging device 302 in this case has a configuration similar to that of the example in Figure 15.

[0115] Of course, this technology may also be implemented using an imaging communication device 301 instead of the imaging device 302. A main example of the configuration of the imaging communication device 301 in that case is shown in Figure 17. As shown in Figure 17, the imaging communication device 301 in this case has the same configuration as the imaging device 302 in the example in Figure 16.

[0116] In other words, the configuration shown in Figure 14 (each processing block, etc.) may be implemented in the imaging communication device 301 or in the imaging device 302.

[0117] <Image Processing Flow 1> An example of the imaging process flow performed by the imaging communication device 301 and imaging device 302 configured in this way will be explained with reference to the flowchart in Figure 18. Figure 18 shows an example of the processing flow when the anomaly detection result is presented immediately. Furthermore, the configuration example in Figure 14 will be used for this explanation.

[0118] When the imaging process is started, in step S301, the 3D attitude derivation unit 314 derives the 3D attitude.

[0119] In step S302, the imaging unit 331 captures an image of the subject and generates an image according to the control of the imaging control unit 315.

[0120] In step S303, the anomaly detection unit 316 performs anomaly detection on the captured image.

[0121] In step S304, the data management unit 317 stores the captured image, the anomaly detection result, and the imaging-related information in the storage unit 318, associating them with each other. The storage unit 318 stores this information as it relates to each other.

[0122] In step S305, the detection result presentation processing unit 319 immediately presents the abnormality detection result to the abnormality detection process.

[0123] In step S306, the detection result presentation processing unit 319 determines whether or not to re-image the abnormal image based on requests from the user or the like, which are input via the input unit 322. If it is determined that re-image is necessary, the process proceeds to step S307.

[0124] In step S307, the re-imaging guidance unit 320 performs re-imaging guidance.

[0125] In step S308, the 3D attitude derivation unit 314 derives the 3D attitude.

[0126] In step S309, the imaging unit 331, following guidance from the re-imaging guidance unit 320 and control from the imaging control unit 315, re-imaging under the same imaging conditions as the abnormal image and generating a re-imaging image.

[0127] In step S310, the anomaly detection unit 316 immediately performs anomaly detection on the re-captured image.

[0128] In step S311, the data management unit 317 replaces the original image with the re-captured image and stores it in the storage unit 318. At the same time, the data management unit 317 stores the anomaly detection result and the imaging-related information in association with each other. The storage unit 318 stores this information in association with each other.

[0129] In step S312, the detection result presentation processing unit 319 immediately presents the abnormality detection result for the abnormality detection process. When the processing in step S312 is completed, the process proceeds to step S313. Also, if it is determined in step S306 that re-imaging is not necessary, the process proceeds to step S313.

[0130] In step S313, the imaging control unit 315 determines whether or not to terminate the imaging process. If it is determined not to terminate the imaging process, the process returns to step S301, and the subsequent processes are executed. If it is determined in step S313 to terminate the imaging process, the imaging process is terminated.

[0131] By performing each process in this manner, users can more easily retake abnormal images during photogrammetry imaging.

[0132] <Image Processing Flow 2> Next, an example of the imaging process flow when anomaly detection results are presented non-immediately (after the completion or interruption of the imaging process) will be explained with reference to the flowcharts in Figures 19 and 20.

[0133] In this case, when the imaging process is started, in step S341 of Figure 19, the 3D attitude derivation unit 314 derives the 3D attitude.

[0134] In step S342, the imaging unit 331 captures an image of the subject and generates an image according to the control of the imaging control unit 315.

[0135] In step S343, the anomaly detection unit 316 performs anomaly detection on the captured image.

[0136] In step S344, the data management unit 317 stores the captured image, the anomaly detection result, and the imaging-related information in the storage unit 318, associating them with each other. The storage unit 318 stores this information as it relates to each other.

[0137] In step S345, the imaging control unit 315 determines whether to terminate or interrupt the imaging operation. If it is determined that the imaging operation should neither be terminated nor interrupted, the process returns to step S341 and repeats the subsequent processes. If it is determined in step S345 that the imaging operation should be terminated or interrupted, the process proceeds to step S346.

[0138] In step S346, the detection result presentation processing unit 319 presents the abnormality detection result after the imaging operation is completed or interrupted. Once the processing in step S346 is completed, the process proceeds to Figure 20.

[0139] In step S351 of Figure 20, the detection result presentation processing unit 319 determines whether or not to re-image the abnormal image based on requests from the user or the like, which are input via the input unit 322. If it is determined that re-image is necessary, the process proceeds to step S352.

[0140] In step S352, the re-imaging guidance unit 320 starts the re-imaging operation and performs re-imaging guidance.

[0141] In step S353, the 3D attitude derivation unit 314 derives the 3D attitude.

[0142] In step S354, the imaging unit 331, following guidance from the re-imaging guidance unit 320 and control from the imaging control unit 315, re-imaging under the same imaging conditions as the abnormal image and generating a re-imaging image.

[0143] In step S355, the anomaly detection unit 316 immediately performs anomaly detection on the re-captured image.

[0144] In step S356, the data management unit 317 replaces the original image with the re-captured image and stores it in the storage unit 318. At the same time, the data management unit 317 stores the anomaly detection result and the imaging-related information in association with each other. The storage unit 318 stores this information in association with each other.

[0145] In step S357, the imaging control unit 315 determines whether to terminate or interrupt the re-imaging operation. If it is determined that the re-imaging operation should neither be terminated nor interrupted, the process returns to step S352 and repeats the subsequent processes. If it is determined in step S357 that the imaging operation should be terminated or interrupted, the process proceeds to step S358.

[0146] In step S358, the detection result presentation processing unit 319 presents the abnormality detection result after the imaging operation is completed or interrupted. Once the processing in step S358 is completed, the process proceeds to step S359. Also, if it is determined in step S351 that re-imaging is not necessary, the process proceeds to step S359.

[0147] In step S359, the imaging control unit 315 determines whether or not to terminate the imaging process. If it is determined that the imaging process should not be terminated, the process returns to step S341 in Figure 19, and the subsequent processes are executed. Alternatively, if it is determined in step S359 in Figure 20 that the imaging process should be terminated, the imaging process is terminated.

[0148] By performing each process in this manner, users can more easily retake abnormal images during photogrammetry imaging.

[0149] <5. Addendum> <Computer> The series of processes described above can be executed by hardware or by software. When the series of processes are executed by software, the programs that make up that software are installed on a computer. Here, "computer" includes computers built into dedicated hardware, as well as general-purpose personal computers, for example, that can perform various functions by installing various programs.

[0150] Figure 21 is a block diagram showing an example of the hardware configuration of a computer that executes the series of processes described above by a program.

[0151] In the computer 1900 shown in Figure 21, the CPU (Central Processing Unit) 1901, ROM (Read Only Memory) 1902, and RAM (Random Access Memory) 1903 are interconnected via a bus 1904.

[0152] Bus 1904 is also connected to an input / output interface 1910. The input / output interface 1910 is connected to an input unit 1911, an output unit 1912, a storage unit 1913, a communication unit 1914, and a drive 1915.

[0153] The input unit 1911 consists of, for example, a keyboard, mouse, microphone, touch panel, and input terminals. The output unit 1912 consists of, for example, a display, speaker, and output terminals. The storage unit 1913 consists of, for example, a hard disk, RAM disk, and non-volatile memory. The communication unit 1914 consists of, for example, a network interface. The drive 1915 drives removable media 1921 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory.

[0154] In a computer configured as described above, the CPU 1901 loads, for example, a program stored in the memory unit 1913 into the RAM 1903 via the input / output interface 1910 and the bus 1904, and executes it, thereby performing the series of processes described above. The RAM 1903 also stores data necessary for the CPU 1901 to perform various processes as appropriate.

[0155] The program to be executed by the computer can be recorded and applied to a removable medium 1921, such as a package media. In this case, the program can be installed to the storage unit 1913 via the input / output interface 1910 by inserting the removable medium 1921 into the drive 1915.

[0156] Furthermore, this program can also be provided via wired or wireless transmission media such as local area networks, the internet, or digital satellite broadcasting. In that case, the program can be received by the communication unit 1914 and installed in the storage unit 1913.

[0157] Additionally, this program can be pre-installed on ROM1902 or memory unit 1913.

[0158] <Applicability of this technology> This technology can be applied to any configuration. For example, this technology can be applied to various electronic devices.

[0159] Furthermore, this technology can also be implemented as part of a device, such as a processor (e.g., a video processor) as a system LSI (Large Scale Integration), a module using multiple processors (e.g., a video module), a unit using multiple modules (e.g., a video unit), or a set with additional functions added to a unit (e.g., a video set).

[0160] Furthermore, this technology can also be applied to network systems composed of multiple devices. For example, this technology may be implemented as cloud computing, where multiple devices share and collaborate on processing via a network. For example, this technology may be implemented in a cloud service that provides image (video) related services to any terminal such as computers, AV (Audio Visual) equipment, portable information processing terminals, and IoT (Internet of Things) devices.

[0161] In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all components are located in the same enclosure. Therefore, multiple devices housed in separate enclosures and connected via a network, and a single device containing multiple modules within a single enclosure, are both considered systems.

[0162] <Fields and applications where this technology can be applied> Systems, devices, and processing units incorporating this technology can be used in any field, such as transportation, healthcare, security, agriculture, livestock farming, mining, beauty, factories, home appliances, weather, and nature monitoring. Furthermore, their applications are entirely arbitrary.

[0163] <Other> In this specification, "flag" refers to information used to identify multiple states, and includes not only information used to identify two states, true (1) or false (0), but also information capable of identifying three or more states. Therefore, the values ​​that this "flag" can take are, for example, two values, 1 / 0, or three or more values. In other words, the number of bits that constitute this "flag" is arbitrary, and can be one bit or multiple bits. Furthermore, identification information (including flags) can be included not only in the form of the identification information itself in the bitstream, but also in the form of differential information of the identification information relative to a certain reference information in the bitstream. Therefore, in this specification, "flag" and "identification information" include not only the information itself, but also differential information relative to the reference information.

[0164] Furthermore, various types of information (metadata, etc.) related to encoded data (bitstream) may be transmitted or recorded in any form as long as they are associated with the encoded data. Here, the term "associate" means, for example, making it possible to use (link) one piece of data when processing the other. In other words, associated data may be combined into a single piece of data, or they may be individual pieces of data. For example, information associated with encoded data (image) may be transmitted on a different transmission path than the encoded data (image). Also, for example, information associated with encoded data (image) may be recorded on a different recording medium (or a different recording area on the same recording medium) than the encoded data (image). Note that this "association" does not have to be with the entire data, but only with a part of the data. For example, an image and the information corresponding to that image may be associated with each other in any unit, such as multiple frames, one frame, or a part within a frame.

[0165] In this specification, terms such as "combine," "multiplex," "add," "integrate," "include," "store," "insert," "insert," and "place" mean combining multiple things into one, such as combining encoded data and metadata into a single data, and represent one method of "associating" as described above.

[0166] Furthermore, the embodiments of this technology are not limited to those described above, and various modifications are possible without departing from the gist of this technology.

[0167] For example, the configuration described as a single device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, the configurations described above as multiple devices (or processing units) may be combined and configured as a single device (or processing unit). Furthermore, it is also possible to add configurations other than those described above to the configuration of each device (or each processing unit). In addition, if the overall system configuration and operation are substantially the same, a part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0168] Furthermore, for example, the program described above may be executed on any device. In that case, the device should have the necessary functions (such as functional blocks) and be able to obtain the necessary information.

[0169] Furthermore, for example, each step of a flowchart may be executed by one device, or it may be divided among multiple devices. Additionally, if a single step includes multiple processes, these processes may be executed by one device, or they may be divided among multiple devices. In other words, multiple processes included in a single step can be executed as multiple steps. Conversely, processes described as multiple steps can be combined and executed as a single step.

[0170] Furthermore, for example, a program executed by a computer may be structured so that the steps of the program are executed chronologically in the order described herein, or they may be executed in parallel or individually at necessary times, such as when a call is made. In other words, the steps may be executed in an order different from the order described above, as long as no inconsistencies arise. Moreover, the steps of this program may be executed in parallel with the processing of other programs, or in combination with the processing of other programs.

[0171] Furthermore, for example, multiple technologies relating to this technology can be implemented independently, as long as they do not create a contradiction. Of course, any multiple technologies can also be implemented in combination. For example, some or all of the technologies described in one embodiment can be implemented in combination with some or all of the technologies described in another embodiment. Also, some or all of the above-mentioned technologies can be implemented in combination with other technologies not mentioned above.

[0172] Furthermore, this technology can also be configured as follows. (1) An anomaly detection unit that immediately performs anomaly detection on the captured image when generating an image for generating three-dimensional shape information that represents the three-dimensional shape of a 3D object, A storage unit that stores the captured image and the result of the anomaly detection in association with each other. An information processing device equipped with the following features. (2) The abnormality detection unit detects excessive blurring and shaking in the captured image as abnormalities. (1) The information processing device described above. (3) The storage unit further stores imaging-related information relating to the imaging that generated the captured image, as well as the captured image. The information processing device described in (1) or (2). (4) The imaging-related information includes information regarding the position and orientation of the camera at the time of imaging. (3) The information processing device described above. (5) The imaging-related information includes imaging parameters applied to the imaging. (3) or (4) the information processing device described above. (6) Further comprising a detection result presentation processing unit that immediately presents the result of the anomaly detection in response to the anomaly detection. An information processing device as described in any of (1) to (5). (7) The system further comprises a re-imaging guidance request receiving unit that receives a request for guidance to re-image under the same imaging conditions as the captured image in which an anomaly was detected due to the anomaly detection. (6) The information processing device described above. (8) The system further comprises a detection result presentation processing unit that presents the abnormality detection results stored in the storage unit after the completion or interruption of the imaging operation that generates the captured image. An information processing device as described in any of (1) to (7). (9) The detection result presentation processing unit presents the results of the anomaly detection superimposed on the real space. (8) The information processing device described above. (10) The detection result presentation processing unit presents the result of the anomaly detection superimposed on the real space as seen from the current camera viewpoint. (9) The information processing device described above. (11) The detection result presentation processing unit presents the result of the anomaly detection superimposed on the virtual space. An information processing device as described in any of (8) to (10). (12) The detection result presentation processing unit presents the results of the anomaly detection superimposed on the virtual space viewed from a predetermined overhead viewpoint. (11) The information processing device described above. (13) The detection result presentation processing unit presents the result of the anomaly detection superimposed on the virtual space as seen from the viewpoint of a virtual camera. The information processing device described in (11) or (12). (14) The detection result presentation processing unit presents the result of the anomaly detection in the list of captured images. An information processing device as described in any of (8) to (13). (15) The system further comprises a re-imaging guidance request receiving unit that receives a request for guidance to re-image under the same imaging conditions as the captured image in which an anomaly was detected due to the anomaly detection. An information processing device as described in any of (8) to (14). (16) The system further comprises a re-imaging guidance processing unit that guides the re-imaging of the captured image under the same imaging conditions as the captured image in which the anomaly was detected by the anomaly detection. An information processing device as described in any of (1) to (15). (17) The re-imaging guidance processing unit presents the area to be re-imaged superimposed on the real space. (16) The information processing device described above. (18) The re-imaging guidance processing unit presents the area to be re-imaged superimposed on the real space as seen from the current camera's viewpoint. (17) The information processing device described above. (19) Further comprising a re-imaging instruction receiving unit that receives an instruction to perform the re-imaging. An information processing device as described in any of (16) to (18). (20) The storage unit stores a re-captured image obtained by re-capturing under the same imaging conditions as the captured image in which the abnormality was detected, replacing the captured image in which the abnormality was detected. An information processing device as described in any of (1) to (19). (21) The storage unit applies the file name of the captured image in which the abnormality was detected as the file name of the re-captured image. (20) The information processing device described above. (22) The storage unit applies the metadata of the captured image in which the abnormality was detected as the metadata of the re-captured image. The information processing device described in (20) or (21). (23) Further comprising an imaging unit that captures an image of a subject and generates the captured image, The anomaly detection unit performs anomaly detection on the captured image immediately in response to the image captured by the imaging unit. An information processing device as described in any of (1) to (22). (24) Immediately perform anomaly detection on the captured image when generating an image for generating three-dimensional shape information that represents the three-dimensional shape of a 3D object, The captured image and the result of the anomaly detection are stored in association with each other. Information processing methods including (25) Immediately perform anomaly detection on the captured image when generating an image for which three-dimensional shape information representing the three-dimensional shape of a 3D object is generated, The captured image and the result of the anomaly detection are stored in association with each other. A program that causes a computer to perform a process that includes [a specific action]. [Explanation of Symbols]

[0173] 301 Imaging communication device, 302 Imaging device, 311 Depth sensor, 312 Imaging unit, 313 IMU, 314 3D attitude derivation unit, 315 Imaging control unit, 316 Anomaly detection unit, 317 Data management unit, 318 Storage unit, 319 Detection result presentation processing unit, 320 Re-imaging guidance unit, 321 Display unit, 322 Input unit, 331 Imaging unit, 1900 Computer

Claims

1. An anomaly detection unit that immediately performs anomaly detection on the captured image when generating an image for generating three-dimensional shape information that represents the three-dimensional shape of a 3D object, A storage unit that stores the captured image and the result of the anomaly detection in association with each other. An information processing device equipped with the following features.

2. The anomaly detection unit detects excessive blurring and shaking in the captured image as an anomaly. The information processing apparatus according to claim 1.

3. The storage unit also stores imaging-related information relating to the imaging that generated the captured image, associating it with the captured image. The information processing apparatus according to claim 1.

4. The system further includes a detection result presentation processing unit that immediately presents the results of the anomaly detection in response to the anomaly detection. The information processing apparatus according to claim 1.

5. The system further includes a re-imaging guidance request receiving unit that receives a request for guidance on re-imaging under the same imaging conditions as the captured image in which an anomaly was detected. The information processing apparatus according to claim 4.

6. The system further includes a detection result presentation processing unit that presents the abnormality detection results stored in the storage unit after the completion or interruption of the imaging operation that generates the aforementioned captured image. The information processing apparatus according to claim 1.

7. The detection result presentation processing unit presents the results of the anomaly detection superimposed on the real space. The information processing apparatus according to claim 6.

8. The detection result presentation processing unit presents the results of the anomaly detection superimposed on the real space as seen from the current camera's viewpoint. The information processing apparatus according to claim 7.

9. The detection result presentation processing unit presents the results of the anomaly detection superimposed on the virtual space. The information processing apparatus according to claim 7.

10. The detection result presentation processing unit presents the results of the anomaly detection superimposed on the virtual space viewed from a predetermined overhead perspective. The information processing apparatus according to claim 9.

11. The detection result presentation processing unit presents the result of the anomaly detection by superimposing it onto the virtual space as viewed from the viewpoint of a virtual camera. The information processing apparatus according to claim 9.

12. The detection result presentation processing unit presents the results of the anomaly detection in the list of captured images. The information processing apparatus according to claim 6.

13. The system further includes a re-imaging guidance request receiving unit that receives a request for guidance on re-imaging under the same imaging conditions as the captured image in which an anomaly was detected. The information processing apparatus according to claim 6.

14. The system further includes a re-imaging guidance processing unit that guides the user to re-image under the same imaging conditions as the captured image in which the anomaly was detected. The information processing apparatus according to claim 1.

15. The re-imaging guidance processing unit presents the area to be re-imaged superimposed on the real space. The information processing apparatus according to claim 14.

16. The re-imaging guidance processing unit presents the area to be re-imaged by superimposing it onto the real space as seen from the current camera's viewpoint. The information processing apparatus according to claim 15.

17. The system further comprises a re-imaging instruction receiving unit that receives instructions to perform the aforementioned re-imaging. The information processing apparatus according to claim 14.

18. The storage unit stores a re-captured image obtained by re-capturing under the same imaging conditions as the captured image in which the anomaly was detected, replacing the captured image in which the anomaly was detected. The information processing apparatus according to claim 1.

19. The system further includes an imaging unit that captures an image of a subject and generates the captured image, The anomaly detection unit performs anomaly detection on the captured image immediately in response to the image captured by the imaging unit. The information processing apparatus according to claim 1.

20. Immediately upon generating an image for capturing data to generate three-dimensional shape information representing the three-dimensional shape of a 3D object, anomaly detection is performed on the captured image. The captured image and the result of the anomaly detection are stored in association with each other. Information processing methods including

Images