Information processing device and method

Abstract

The present disclosure relates to an information processing device and method capable of estimating a posture at an earlier timing. The present invention determines whether to supply a captured image or feature amount information generated from the captured image to a device that derives posture information indicating the posture of a camera corresponding to a captured image, and supplies the captured image or the feature amount information to the device according to the determination. The present invention acquires a captured image or feature amount information generated from the captured image, and uses the acquired captured image or feature amount information to derive posture information indicating the posture of a camera corresponding to the captured image. The present disclosure can be applied to, for example, an information processing device, an image-capturing device, an image-capturing communication device, and an electronic apparatus, an information processing method, a program, an information processing system or the like.

Description

Information processing device and method

[0001] The present disclosure relates to an information processing device and method, and more particularly to an information processing device and method that enable earlier posture estimation.

[0002] Conventionally, a method known as photogrammetry for 3D compositing of objects having three-dimensional shapes involves capturing images of the object from multiple directions, performing 3D modeling based on the captured images, and generating 3D data (see, for example, Patent Document 1). Photogrammetry involves processing that places a heavy load on the system, such as estimating the camera's orientation corresponding to the captured images. To address this issue, a method has been devised in which the captured images are transferred to a server and 3D modeling is performed there. Another method is incremental 3D compositing, in which 3D modeling is performed sequentially as the captured images are supplied.

[0003] JP 2018-63693 A

[0004] However, the captured image data is large and can take a long time to transmit to the server, which can increase the delay time until the posture estimation result is obtained.

[0005] The present disclosure has been made in view of such circumstances, and aims to enable posture estimation at an earlier stage.

[0006] An information processing device according to one aspect of the present technology is an information processing device that includes a decision unit that decides whether to supply a captured image or feature information generated from the captured image to a device that derives posture information indicating the posture of a camera corresponding to the captured image, and a supply unit that supplies the captured image or the feature information to the device in accordance with the decision.

[0007] An information processing method according to one aspect of the present technology is an information processing method that determines whether to supply a captured image or feature information generated from the captured image to a device that derives posture information indicating the posture of a camera corresponding to the captured image, and then supplies the captured image or the feature information to the device in accordance with the determination.

[0008] An information processing device according to another aspect of the present technology is an information processing device including: an acquisition unit that acquires a captured image or feature information generated from the captured image; and an attitude information derivation unit that uses the acquired captured image or the feature information to derive attitude information indicating the attitude of a camera corresponding to the captured image.

[0009] An information processing method according to another aspect of the present technology is an information processing method that acquires a captured image or feature information generated from the captured image, and derives posture information indicating the posture of a camera corresponding to the captured image using the acquired captured image or feature information.

[0010] In an information processing device and method according to one aspect of the present technology, a decision is made as to whether to supply the captured image or feature information generated from the captured image to a device that derives posture information indicating the posture of a camera corresponding to the captured image, and the captured image or feature information is supplied to the device in accordance with the decision.

[0011] In the information processing device and method according to another aspect of the present technology, a captured image or feature information generated from the captured image is acquired, and posture information indicating the posture of a camera corresponding to the captured image is derived using the acquired captured image or feature information.

[0012] 19 is a diagram for explaining an overview of photogrammetry. FIG. 19 is a diagram for explaining an example of the flow of photogrammetry processing. FIG. 20 is a diagram for explaining an example of posture estimation. FIG. 21 is a diagram for explaining an example of a sequence. FIG. 22 is a diagram for explaining increment processing. FIG. 23 is a diagram for explaining an overview of control. FIG. 24 is a diagram for explaining an example of transmission control. FIG. 25 is a diagram for explaining an example of a GUI for accepting user instructions. FIG. 26 is a diagram for explaining an example of priority setting. FIG. 27 is a diagram for explaining an example of point cloud evaluation. FIG. 28 is a diagram for explaining an example of a comparison of transmission data amounts. FIG. 29 is a block diagram showing an example of the main configuration of a 3D synthesis system. FIG. 29 is a block diagram showing an example of the main configuration of a terminal device. FIG. 29 is a block diagram showing an example of the main configuration of a transmission information generation unit. FIG. 29 is a block diagram showing an example of the main configuration of a posture estimation unit. FIG. 29 is a block diagram showing an example of the main configuration of a server. FIG. 29 is a block diagram showing an example of the main configuration of a posture estimation unit. FIG. 29 is a flowchart for explaining an example of the flow of 3D synthesis system processing. FIG. 29 is a flowchart for explaining an example of the flow of transmission control processing. FIG. 29 is a flowchart for explaining an example of the flow of transmission control processing. FIG. 29 is a flowchart for explaining an example of the flow of transmission control processing. FIG. 29 is a flowchart for explaining an example of the flow of transmission information determination processing. FIG. 29 is a flowchart for explaining an example of the flow of transmission information determination processing. FIG. 29 is a flowchart for explaining an example of the flow of transmission information determination processing. 12 is a flowchart continuing from FIG. 11 , illustrating an example of the flow of a 3D modeling process. FIG. 13 is a flowchart illustrating an example of the flow of a transmission control process. FIG. 14 is a flowchart illustrating an example of the flow of a posture estimation process. FIG. 15 is a block diagram showing an example of the main configuration of a 3D synthesis system. FIG. 16 is a block diagram showing an example of the main configuration of a 3D synthesis system. FIG. 17 is a block diagram showing an example of the main configuration of a 3D synthesis system. FIG. 18 is a block diagram showing an example of the main configuration of a 3D synthesis device. FIG. 19 is a block diagram showing an example of the main configuration of a computer.

[0013] Hereinafter, modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described. The description will be made in the following order: 1. 3D modeling 2. Transmission control 3. First embodiment (3D composition system) 4. Other configuration examples 5. Supplementary notes

[0014] <1. 3D Modeling> <Photogrammetry> Conventionally, a method for generating (reconstructing) a model of a three-dimensional shape of an object (also referred to as a 3D object in this specification) has been known as photogrammetry, in which the 3D object is imaged from multiple directions and 3D data is generated based on the multiple captured images. Note that in this specification, generating a model of the three-dimensional shape of a 3D object is also referred to as 3D modeling.

[0015] Photogrammetry is a technique for reconstructing highly accurate three-dimensional models using the principles of triangulation from multiple images taken from various viewpoints. In this specification, the term "accuracy" in reference to 3D data (3D models) refers not only to the reproducibility (accuracy, definition, etc.) of the three-dimensional shape of the target 3D object, but also to the reproducibility (accuracy, definition, etc.) of the texture applied to the surface of the 3D model. For example, cameras 11-1 to 11-5 shown in FIG. 1 capture images of a 3D object 10 from multiple viewpoints to obtain multiple captured images. These captured images are then used to perform processes called SfM (Structure from Motion) and MVS (Multiview Stereo), followed by post-processing such as meshing and texturing to generate 3D data 15.

[0016] In SfM, for example, corresponding points are searched for between captured images, the position and orientation of the camera are derived using epipolar constraints, and the positions of each corresponding point in three-dimensional space are identified by triangulation based on the camera position and orientation. In this specification, these points in three-dimensional space are also referred to as three-dimensional points. In other words, three-dimensional points corresponding to each corresponding point are identified. The entire three-dimensional point cloud identified in this way is then optimized by bundle adjustment.

[0017] In MVS, for example, a dense corresponding point search is further performed using the 3D point group derived as described above, and 3D points are added.

[0018] More specifically, the process is performed according to the procedure shown in FIG. 2 . In imaging 21, a subject (3D object) is captured and a captured image is generated. In image reduction 22, the captured image is reduced to a predetermined size (resolution). For example, a captured image of 4K size (3840 x 2160 pixels) is reduced to full HD size (1920 x 1080 pixels). In feature point extraction 23, feature points are extracted from the captured image. For example, assume that there are two captured images 31 and 32 as shown in the upper part of FIG. 3 . Pixels (P11, P12, P13) having predetermined features in captured image 31 are extracted as feature points (P11, P12, P13). Similarly, pixels (P21, P22, P23) having predetermined features in captured image 32 are extracted as feature points (P21, P22, P23).

[0019] Feature point matching 24 matches feature points extracted from each captured image. That is, corresponding feature points between the captured images (feature points corresponding to the same three-dimensional position) are detected. In this specification, these corresponding feature points between the captured images are also referred to as corresponding points. For example, as shown in the middle of FIG. 3, matching is performed between captured image 31 and captured image 32, and a correspondence relationship between feature points as indicated by the dotted double-headed arrows in the figure is determined. In the example of FIG. 3, feature points P11 and P21 correspond to each other, feature points P12 and P22 correspond to each other, and feature points P13 and P23 correspond to each other.

[0020] The posture estimation 25 estimates the posture of the camera corresponding to each captured image using the corresponding points. That is, camera posture information 26 at the time of capturing each captured image is derived. In this specification, the "posture of the camera" includes all information for obtaining the angle of view of the captured image, such as the position, orientation, and capturing range of the camera. In other words, the "posture of the camera" includes both the internal parameters (optical parameters) and external parameters (rotation, translation) of the camera. Furthermore, the "posture of the camera corresponding to the captured image" refers to the posture of the camera when the captured image is generated (at the time of capturing the image). Furthermore, the "camera posture information" refers to information indicating the posture of the camera. In other words, the "camera posture information at the time of capturing the captured image" refers to information indicating the posture of the camera when the captured image is generated (at the time of capturing the image). Therefore, the "camera posture information at the time of capturing the captured image" is also referred to as the "camera posture information corresponding to the captured image." For example, as shown in the lower part of Figure 3, the epipolar line of feature point P11 in captured image 31 and the epipolar line of feature point P21 in captured image 32, which are indicated by dotted lines in the figure, are obtained, and epipolar lines are similarly obtained for other corresponding points, and these are used to estimate the orientation of camera C1 corresponding to captured image 31 and the orientation of camera C2 corresponding to captured image 32. Camera orientation information 26 generated in this way by orientation estimation 25 may be output data. For example, the camera orientation information 26 may be stored or used for other processing.

[0021] Depth image estimation by stereo matching 27 uses the posture estimation results (camera posture information 26) to perform stereo matching of the captured image, and three-dimensional points corresponding to each corresponding point are identified and optimized by bundle adjustment.

[0022] The depth value fusion process fuses the three-dimensional point clouds derived as described above, generating point cloud data 29 that represents the three-dimensional structure of the 3D object.

[0023] As described above, photogrammetry performs a global optimization calculation called bundle adjustment to minimize errors, which allows it to obtain very accurate results, but it also places a heavy burden on the calculations.Furthermore, because it is based on geometric calculations rather than physical measurements, in principle, the higher the resolution of the images used, the more accurate the model can be restored.

[0024] Because photogrammetry involves such heavy processing loads, executing such processing on low-performance information processing devices (e.g., cameras, smartphones, tablet PCs, laptops, etc.) used at the imaging site could result in impractical processing times. To address this issue, a method was devised in which captured images are transferred to a server and 3D modeling is performed on a high-performance server. This method allows for faster 3D modeling, making it possible to obtain camera posture information and 3D data more quickly than with the above-mentioned methods.

[0025] <Incremental 3D Composition> As described above, the process of capturing images of an object and using the captured images to perform 3D modeling is generally performed according to the flow shown in flow 41 in Figure 4. In other words, in this case, the user travels to the location of the object and captures the image of the object. Once the captured images necessary for 3D modeling are collected, the user stops capturing the images and returns to their home or office, where they use a stationary, high-spec computer to composite and edit the 3D model. Therefore, if a capture is missed (for example, if there are insufficient valid captured images and a high-precision 3D model cannot be obtained for a certain area), the user must return to the site and restart the capturing process, which is a cumbersome process.

[0026] Therefore, for example, a method has been devised in which 3D modeling is performed for feedback during imaging work, separate from the final 3D modeling (e.g., 3D model synthesis in flow 41). That is, as shown in flow 42 of Figure 4, a method has been devised in which 3D modeling (3D model synthesis) is performed in parallel with imaging work. Naturally, since imaging work is ongoing, not all captured images have been acquired. Therefore, in this 3D modeling, processing is performed sequentially using the captured images acquired up to that point. This allows the user to check (partial) 3D modeling results (part of the 3D model) before generating the number of captured images required to construct the entire 3D model. Therefore, during imaging work, the user can check whether there are any "missing images" (areas where there are insufficient images) in the previously captured areas. If a "missing image" is identified, the user can redo the imaging of that area. In this way, the user can use the 3D model at the imaging site during imaging work to navigate the imaging work, etc. That is, the user can easily and quickly check the imaging results (3D model) without having to travel to their home, office, etc., as in the case of flow 41. Therefore, the user can more easily perform imaging for 3D modeling.

[0027] In other words, this sequential 3D modeling (3D modeling by incremental processing) can be performed multiple times during the imaging process. As the imaging process progresses and the number of captured images increases, a 3D model with a wider range or higher resolution can be obtained. For example, as shown in FIG. 5 , a 3D model 50 obtained in the first processing step only covers the area indicated by the rectangle 51. With each subsequent processing step, the number of captured images used increases, resulting in a larger 3D model 50, as indicated by the rectangles 52, 53, and 54, respectively. In this specification, this sequential 3D modeling is also referred to as incremental 3D synthesis.

[0028] For example, by displaying the 3D model 50 shown in the square 51, the user can check the 3D modeling results for this displayed portion. For example, the user can check whether there are any "missing images" for this portion. In other words, an incomplete 3D model is fed back, but the user can still use the 3D model even if it is incomplete. In this way, by sequentially feeding back the 3D model 50 obtained by incremental 3D synthesis, the user can check the 3D modeling results before the 3D model is completed. Therefore, typically, the user can (sequentially) check the 3D modeling results at the site where the imaging work is being performed before finishing the imaging work. This makes it easier for the user to perform imaging for 3D modeling than when the 3D modeling results cannot be referenced at all during the imaging work, as in flow 42 (i.e., when the user checks the 3D modeling results after returning home or the office).

[0029] The method for this incremental 3D compositing may be the same as that used for the final 3D modeling (e.g., the 3D model compositing in flow 41). For example, photogrammetry processing may be applied. By applying a method similar to that used for the final 3D modeling, a 3D model that is substantially the same as the processing result of the final 3D modeling can be fed back. Note that the incremental 3D compositing may be performed using exactly the same processing as that used for the final 3D modeling, or some processing may be omitted or simplified to shorten processing time. Furthermore, the amount of information in the captured image used for the incremental 3D compositing may be reduced. For example, compressed images or thumbnail images may be applied to the incremental 3D compositing. As described above, since the 3D model obtained by incremental 3D compositing is data intended to be used primarily temporarily for confirmation work, etc., high processing speed may be emphasized over the accuracy (resolution) of the model.

[0030] In this way, in 3D modeling based on captured images, such as photogrammetry, there is a one-time method in which 3D modeling is performed after all captured images are collected, as well as the incremental 3D synthesis method described above.

[0031] <Increased Processing Time> However, captured images have a large amount of data, and transmission to a server can take a long time. In particular, when the bandwidth of a transmission path is variable, such as the Internet, narrowing the bandwidth can increase the time required to transmit the captured images. This increase in the time required to transmit the captured images can increase the delay time until the pose estimation result is obtained. In particular, when performing incremental 3D synthesis as described above, it is highly desirable to estimate the pose as quickly as possible.

[0032] <2. Transmission Control> <Determination of Transmission Information> Therefore, the type of information to be transmitted to the server that derives camera posture information corresponding to the captured image is controlled. More specifically, it is determined whether to supply the captured image to the server or to supply feature amount information generated from the captured image, and the determined information is supplied to the server as transmission information. The feature amount information is information that indicates feature amounts for predetermined features of the captured image.

[0033] 6 performs image capture and feature amount information generation, and its control unit 61A determines whether to transmit the captured image to the server 62 or the feature amount information to the server 62. The terminal device 61 supplies the determined option to the server 62 as transmission information.

[0034] The server 62 uses the supplied transmission information (captured image or feature amount information) to derive attitude information indicating the attitude of the camera corresponding to the captured image. At this time, the control unit 62A controls the processing to be executed in accordance with the supplied transmission information. That is, the control unit 62A executes the remaining processing for deriving attitude information that has not been executed by the terminal device 61 in accordance with the acquired transmission information. The control of which processing to execute may be based on any information. For example, flag information indicating what information will be transmitted as transmission information may be supplied from the terminal device 61, and the server 62 (control unit 62A) may determine what information the transmission information is based on the flag information, identify the remaining processing that has not been executed by the terminal device 61, and control the server 62 to execute the identified processing. Alternatively, the server 62 (control unit 62A) may determine what information the transmission information is based on the data volume of the transmission information, identify the remaining processing that has not been executed by the terminal device 61, and control the server 62 to execute the identified processing. Alternatively, the server 62 (controller 62A) may grasp the type of information to be transmitted based on the extension of the information, identify the remaining processes that have not been executed in the terminal device 61, and control the terminal device 61 to execute the identified processes. Of course, other methods may also be used.

[0035] The server 62 may derive the posture information sequentially or collectively. That is, the server 62 may sequentially derive the posture information that can be derived using the acquired captured images or feature amount information. Alternatively, the server 62 may acquire all the captured images or feature amount information and then derive all the posture information collectively.

[0036] Generally, the amount of data in the feature information generated from a captured image is smaller than that in the captured image. Therefore, the time required to transmit the feature information is shorter than that in the captured image. However, the server 62 has higher performance and a faster processing speed than the terminal device 61. Therefore, by appropriately controlling whether to supply the captured image or the feature information depending on the situation, the increase in the time required for transmission can be suppressed. Therefore, the server can perform posture estimation more quickly.

[0037] The transmission path between the terminal device 61 and the server 62 may be configured by any communication path. For example, it may include a wired communication path, a wireless communication path, or both. For example, this transmission path may include a communication network such as the Internet, a LAN (Local Area Network), a WAN (Wide Area Network), or Wi-Fi. Furthermore, this transmission path may include a communication path such as a USB (Universal Serial Bus) (registered trademark) or an HDMI (High-Definition Multimedia Interface) (registered trademark). Furthermore, this transmission path may include an information processing device such as a server or a relay device. Furthermore, this transmission path may include multiple networks.

[0038] The server 62 may have any physical configuration. For example, the server 62 may be a personal computer owned by a user (photographer), or may be a server owned by a company that provides 3D modeling services. The server 62 may be composed of a single information processing device, or may be a system composed of multiple information processing devices. For example, the server 62 may be a virtual server. The server 62 may also be an entity with no specified physical configuration, such as so-called cloud computing (cloud service).

[0039] The feature information may be any information. For example, the feature information may include feature points, which are pixels that exhibit predetermined features of the captured image. That is, the terminal device 61 that transmits the transmission information may extract feature points from the captured image, and transmit feature information including the feature points to the server 62 as transmission information. The feature information may also include a matching result of the feature points. That is, the terminal device 61 may match feature points extracted from the captured image, and transmit feature information including the matching result to the server 62 as transmission information. The feature information may also include intermediate data of posture estimation using the feature point matching result. That is, the terminal device 61 may perform part of posture estimation using the feature point matching result, and transmit feature information including the obtained intermediate data to the server 62 as transmission information.

[0040] It is also possible to transmit only either the captured image or the feature information for each captured image. In this case, when it is determined that the captured image is to be transmitted, the feature information corresponding to the captured image is not transmitted.

[0041] Furthermore, for each captured image, both the captured image and feature amount information may be transmitted. For example, a captured image whose feature amount information has been transmitted first may be transmitted later (when it is determined that the captured image is to be transmitted). Furthermore, when it is determined that a captured image is to be transmitted, the feature amount information obtained from the captured image may be transmitted together with the captured image.

[0042] Any method for controlling the transmission information by the terminal device 61 (controller 61A thereof) may be used. For example, as shown in the first to third rows from the top of the table in FIG. 7 , the terminal device 61 (controller 61A thereof) may control based on the bandwidth of the transmission path. For example, when the transmission path has a bandwidth wider than a predetermined standard, the terminal device 61 (controller 61A thereof) may determine to supply the captured image to the server 62 as transmission information. Furthermore, when the transmission path has a bandwidth narrower than a predetermined standard, the terminal device 61 (controller 61A thereof) may determine to supply feature information to the server 62 as transmission information. Furthermore, when the transmission path is non-bandwidth (e.g., disconnected, unable to communicate, etc.), the terminal device 61 (controller 61A thereof) may determine not to transmit the image information and feature information to the server 62, but to perform the remaining processing in the terminal device 61. In other words, in this case, the camera posture information is derived in the terminal device 61. Note that this non-bandwidth may include not only a non-connected state but also an extremely narrowband state in which feature information cannot be substantially transmitted.

[0043] Furthermore, instead of deriving the camera posture information in the terminal device 61, the camera posture information may be derived in an information processing device 63 separate from the server 62. That is, the captured image or feature information may be supplied to the information processing device 63, and the camera posture information may be derived. The transmission path between the terminal device 61 and the information processing device 63 may be configured by any communication path, at least a portion of which is different from the transmission path between the terminal device 61 and the server 62. For example, the transmission path may include a wired communication path, a wireless communication path, or both. For example, the transmission path may include a communication network such as the Internet, a LAN, a WAN, or Wi-Fi. The transmission path may also include a communication path such as USB or HDMI. The transmission path may also include an information processing device such as a server or a relay device. The transmission path may also include multiple networks.

[0044] The information processing device 63 may be any device that can perform pose estimation in the same way as the server 62. For example, the information processing device 63 may be a personal computer owned by a user (photographer), such as a so-called on-premise system. The information processing device 63 may also be an information processing device owned by a company that provides 3D modeling services, separate from the company that owns the server 62. The information processing device 63 may be configured by a single information processing device, or may be a system configured by multiple information processing devices. For example, the information processing device 63 may be a virtual server. The information processing device 63 may also be an entity without a specified physical configuration, such as so-called cloud computing (cloud service).

[0045] Furthermore, it may be possible to control whether posture estimation is performed in the terminal device 61 or in the information processing device 63. When posture estimation is performed in the information processing device 63, the terminal device 61 (control unit 61A) may also control transmission information for this information processing device 63 in the same way as in the case of transmission information for the server 62. In other words, it may be possible to control whether to supply a captured image or feature amount information.

[0046] For example, when the bandwidth is sufficiently wide, transmitting the captured image and having the server 62 extract feature points allows faster processing than deriving feature information in the terminal device 61, and camera posture information can be derived more quickly. Also, when the bandwidth is narrow, transmitting feature information with a small amount of data and having the server 62 perform the remaining processing can prevent an increase in the time required for transmission compared to transmitting the captured image, and camera posture information can be derived more quickly. Also, when there is no bandwidth (when transmission is not possible), performing posture estimation in the terminal device 61 (or information processing device 63) allows camera posture information to be derived more quickly than by waiting.

[0047] Therefore, by controlling the transmission information based on the bandwidth of the transmission path as described above, it is possible to derive camera attitude information earlier in a wider variety of cases.

[0048] Although the above description has been given of controlling transmission information based on the bandwidth of the transmission path, delay time may be applied instead of the bandwidth. For example, the terminal device 61 (or its control unit 61A) may determine to provide the captured image as transmission information to the server 62 when the delay time is smaller than a predetermined first standard. Furthermore, the terminal device 61 (or its control unit 61A) may determine to provide feature information as transmission information to the server 62 when the delay time is medium (when the delay time is equal to or greater than the predetermined first standard and smaller than a predetermined second standard that is greater than the first standard). Furthermore, the terminal device 61 (or its control unit 61A) may determine not to transmit the image information and feature information to the server 62 when the delay time is equal to or greater than a second standard, and instead perform the remaining processing in the terminal device 61. In other words, in this case, camera posture information is derived in the terminal device 61.

[0049] This delay time includes the delay time due to the transmission of the transmission information, and may further include the delay time (processing time) due to the posture estimation performed by the server 62.

[0050] 7, the terminal device 61 (or its control unit 61A) may perform control based on a use case (purpose of the posture information). For example, when the purpose of the posture information is inspection, the terminal device 61 (or its control unit 61A) may determine to supply the captured image to the server 62 as transmission information. When the purpose of the posture information is content generation, the terminal device 61 (or its control unit 61A) may determine to supply the feature information to the server 62 as transmission information.

[0051] As described above, by controlling the transmission information based on the use case (the purpose of the orientation information), it is possible to suppress the transmission of unnecessary information and to prevent unnecessary increases in the time required for transmitting the information. Therefore, it is possible to derive the camera orientation information at an early stage in a wider variety of cases.

[0052] 7 , the information to be transmitted as the transmission information may be controlled manually (by a user instruction, etc.). For example, when the terminal device 61 (control unit 61A) receives a user instruction to supply a captured image, the terminal device 61 may determine to supply the captured image to the server 62 as the transmission information in accordance with the user instruction. When the terminal device 61 (control unit 61A) receives a user instruction to supply feature information, the terminal device 61 may determine to supply the feature information to the server 62 as the transmission information in accordance with the user instruction. When the terminal device 61 (control unit 61A) receives a user instruction not to transmit the transmission information to the server 62, the terminal device 61 (control unit 61A) may determine to perform posture estimation in the terminal device 61 or the information processing device 63 in accordance with the user instruction.

[0053] For example, the terminal device 61 may display an input screen 71 shown in FIG. 8 to accept user instructions. The input screen 71 is a GUI (Graphical User Interface) that accepts user instructions. As shown in FIG. 8 , the input screen 71 displays a guidance message such as, "The bandwidth is XX. The image transmission method takes X minutes, and the feature transmission method takes Y minutes. Which is better? We recommend the △△ method." A GUI button 71A and a GUI button 71B are displayed. The GUI button 71A is a button-type GUI that allows a user to select an image transmission method by pressing the GUI button 71A. In other words, when a user presses the GUI button 71A, a user instruction to provide a captured image to the server 62 as transmission information is accepted. The GUI button 71B is a button-type GUI that allows a user to select a feature transmission method by pressing the GUI button 71B. In other words, when a user presses the GUI button 71B, a user instruction to provide feature information to the server 62 as transmission information is accepted.

[0054] In this way, by controlling the transmission information based on user instructions, it is possible to prevent the time required for transmitting the transmission information from increasing beyond the user's intention, and therefore it is possible to derive camera attitude information earlier in a wider variety of cases.

[0055] Furthermore, as shown in the ninth to eleventh rows from the top of the table in FIG. 7 , the transmission information may be controlled by control information that controls the transmission information. For example, the server 62 may generate control information that controls whether to transmit a captured image or feature information as transmission information and provide the control information to the terminal device 61. The terminal device 61 may then determine what information to provide as transmission information in accordance with the control information. For example, when the terminal device 61 (the control unit 61A) acquires control information from the server 62 instructing it to provide a captured image, it may determine to provide the captured image as transmission information to the server 62 in accordance with the control information. When the terminal device 61 (the control unit 61A) acquires control information from the server 62 instructing it to provide feature information, it may determine to provide the feature information as transmission information to the server 62 in accordance with the control information. When the terminal device 61 (the control unit 61A) acquires control information from the server 62 instructing it not to transmit transmission information to the server 62, it may determine to perform posture estimation in the terminal device 61 or the information processing device 63 in accordance with the control information.

[0056] The server 62 may generate this control information in any manner. For example, the server 62 may measure the bandwidth (or delay time), generate control information in accordance with the measurement results, and supply the control information to the terminal device 61.

[0057] In this way, by controlling the transmission information based on the control information, it is possible to derive the camera attitude information at an early stage in a wider variety of cases.

[0058] Furthermore, the terminal device 61 (control unit 61A thereof) may control the feature amounts (i.e., types of feature points) of the captured image to be supplied as feature amount information. Any information may be used as candidate feature amounts. For example, feature amounts such as SIFT, AKAZE, and ORB may be included. Of course, other feature amounts may also be included. For example, SIFT generally has higher resolution than other feature amounts, but requires a larger amount of data. The feature amount to be applied may be determined according to the differences in the characteristics of each feature amount. In this way, unnecessary increases in the data amount of transmitted information can be suppressed.

[0059] For example, the terminal device 61 (controller 61A thereof) may determine the feature to be applied based on the bandwidth (or delay time) of the transmission path, as in the example of FIG. 7 . Alternatively, the terminal device 61 (controller 61A thereof) may determine the feature to be applied based on a use case (purpose of posture information), as in the example of FIG. 7 . Alternatively, the terminal device 61 (controller 61A thereof) may determine the feature to be applied based on manual selection (user instruction, etc.), as in the example of FIG. 7 . For example, the terminal device 61 may display an input screen 72 shown in FIG. 8 to accept a user instruction. This input screen 72 is a GUI for accepting user instructions. As shown in FIG. 8 , the input screen 72 displays a guidance message such as, "The bandwidth is XX. Feature A takes X minutes, and feature B takes Y minutes. Which is better? I recommend feature △△." A GUI button 72A and a GUI button 72B are displayed. The GUI button 72A is a button-type GUI that allows a user to select feature A by pressing it. That is, when the user presses this GUI button 72A, a user instruction to apply feature A is accepted. The GUI button 72B is a button-type GUI that the user presses to select feature B. That is, when the user presses this GUI button 72B, a user instruction to apply feature B is accepted. Furthermore, (the control unit 61A of) the terminal device 61 may determine the feature to be applied based on control information, as in the example of FIG. 7 . In this way, unnecessary increases in the data volume of transmission information can be suppressed in a wider variety of cases.

[0060] As described above, both the information to be transmitted as transmission information and the feature to be applied may be controlled manually (by user instruction, etc.) In this case, for example, on the terminal device 61, an input screen 71 may be displayed to accept a user instruction to select a transmission method (transmission information), and then an input screen 72 may be displayed to accept a user instruction to select a feature to be applied.

[0061] The feature information may be encoded (compressed). Any encoding method may be used. By doing so, it is possible to suppress an increase in the data amount of the transmission information (feature information).

[0062] In addition, the feature points to be extracted may be limited to a region of interest (ROI (Region of Interest)). That is, a region of interest may be set, feature points of the region of interest may be extracted, and feature amount information of the region of interest may be transmitted. This region of interest may be any region. For example, it may be a part (partial region) of the image region of the captured image. This partial region may be any part of the image region. For example, it may be the center part of the image region or any other part. Alternatively, the captured image may be analyzed to identify a region including a subject, and that region may be used as the region of interest. This image analysis may be performed by any method. For example, image analysis may be performed using AI (Artificial Intelligence). Alternatively, feature points may be extracted from a region of interest including a specified subject in the captured image using an AI-based feature point extraction method specialized for the use case. Alternatively, the region of interest may be within a range in the depth direction. For example, the depth of the subject in the captured image may be detected using an image plane phase difference method or the like, a specified depth range may be used as the region of interest (in the depth direction), and feature points within the region of interest (within the specified depth range) may be extracted.

[0063] By limiting the feature points to a predetermined region of interest in this way, it is possible to suppress an increase in the data amount of transmission information (feature amount information).

[0064] Furthermore, the captured image may be reduced before transmission. For example, when the captured image is used as the transmission information, the terminal device 61 may transmit a reduced image of the captured image to the server 62. The size (resolution) of the captured image generated by imaging may be any size (resolution). Furthermore, the size (resolution) of the captured image after reduction may be any size (resolution). For example, a captured image of 4K size (3840 x 2160 pixels) may be reduced to full HD size (1920 x 1080 pixels). By reducing the captured image before transmission in this way, an increase in the data amount of the transmission information (captured image) can be suppressed.

[0065] Furthermore, the terminal device 61 may select captured images to be transmitted. For example, when a captured image is supplied as transmission information, the terminal device 61 (or its control unit 61A) may select a captured image to be supplied based on the quality of the captured image, and may control so as not to supply captured images of lower quality than a predetermined standard. For example, the terminal device 61 (or its control unit 61A) may determine a captured image in which the number of extracted feature points is less than a predetermined standard as a low-quality captured image and not supply it. In other words, the terminal device 61 may control so as to supply captured images of higher quality than a predetermined standard as transmission information.

[0066] The captured image transmitted by the terminal device 61 may be encoded (compressed). Any encoding method may be used. For example, the terminal device 61 may encode the captured image in the JPEG (Joint Photographic Experts Group) format. Alternatively, the terminal device 61 may encode the captured image in the HEIF (High Efficiency Image File Format) format.

[0067] Furthermore, the terminal device 61 may set a priority order for the captured images to be transmitted. For example, when the captured images are supplied as transmission information, the terminal device 61 (controller 61A) may set a priority order for supplying the captured images and supply the captured images according to the priority order. Any method for setting the priority order may be used. For example, a higher priority order may be set for a captured image with a larger number of newly matching feature points.

[0068] 9 , assume that image capture is performed at camera posture 81, camera posture 82, and camera posture 83, respectively, and captured images are generated. Then, assume that image capture is performed at camera posture 84, generating captured image 86, and then that image capture is performed at camera posture 85, generating captured image 87. Then, assume that feature points are detected in captured image 86 and matched with the feature points of captured images corresponding to camera postures 81, 82, and 83, respectively, to obtain corresponding points as indicated by white circles. Furthermore, assume that feature points are detected in captured image 87 and matched with the feature points of captured images corresponding to camera postures 81, 82, and 83, respectively, to obtain corresponding points as indicated by black circles.

[0069] As shown in FIG. 9 , the range of captured image 86 is included within the ranges of captured images (existing captured images) corresponding to camera poses 81, 82, and 83. Therefore, the corresponding points indicated by white circles are the same as the corresponding points between the existing captured images (the captured images corresponding to camera poses 81, 82, and 83, respectively). In other words, they are known corresponding points. In contrast, the range of captured image 87 also includes areas outside the ranges of captured images (existing captured images) corresponding to camera poses 81, 82, and 83, respectively. Therefore, the corresponding points indicated by black circles are different from the corresponding points between the existing captured images (the captured images corresponding to camera poses 81, 82, and 83, respectively). In other words, they are newly matched feature points.

[0070] Re-detecting known corresponding points does not increase the number of corresponding points available for 3D modeling. In contrast, the more newly matched feature points there are, the greater the number of corresponding points available for 3D modeling. In other words, the more newly matched feature points there are, the greater the contribution to 3D modeling. Therefore, by setting a higher priority for captured images with more newly matched feature points and transmitting them earlier, the contribution to 3D modeling can be improved earlier. In other words, a more highly detailed 3D model can be obtained earlier. For example, in the case of FIG. 9 , by setting a higher priority for captured image 87 than captured image 86 and transmitting captured image 87 before captured image 86, a more highly detailed 3D model can be obtained earlier.

[0071] Note that the server 62 (controller 62A) may set a priority for image processing (deriving posture information, etc.) for captured images or feature amount information, and execute the processing according to the priority. Any method for setting this priority may be used. For example, it may be the same method as the transmission priority setting described above. For example, as described above, a captured image with more newly matching feature points may be given a higher priority, and processed earlier, thereby enabling a more highly detailed 3D model to be obtained earlier.

[0072] When performing pose estimation (up to sparse point cloud generation) on the terminal device 61, the terminal device 61 may omit global optimization (BA) using bundle adjustment for some images. Furthermore, the terminal device 61 may omit part of global optimization (BA) using bundle adjustment for some images. By performing global optimization using bundle adjustment in pose estimation, more accurate camera pose information and a more accurate sparse point cloud can be generated. On the other hand, by omitting global optimization using bundle adjustment or part of it in pose estimation, camera pose information can be generated with less load (reducing the processing load). While this camera pose information is not applicable to 3D modeling due to its low accuracy, it can be used to evaluate a highly accurate sparse point cloud generated by pose estimation with global optimization using bundle adjustment. For example, as shown in FIG. 10, it can be used to determine whether a point cloud is covered (deriving the number of viewpoints from which each point is observed). That is, using both high-precision camera pose information derived by pose estimation involving global optimization using bundle adjustment and low-precision camera pose information derived by pose estimation without global optimization using bundle adjustment, evaluation may be performed based on the number of viewpoints from which each point in the point cloud is observed (the number of observed viewpoints). In the example of FIG. 10 , assume that images are captured from viewpoints 91 to 97, and a point cloud represented by a white circle is generated. Point 98 is observed from viewpoints 91, 92, and 93. Generally, the more viewpoints a point is observed, the higher the reliability of its position information. Therefore, as described above, the number of viewpoints from which each point in the point cloud is observed may be calculated, and the higher the number of viewpoints, the higher the evaluation value (score) of that point may be set. The evaluation result (score) may also be displayed as an image. The score may be displayed in any manner. For example, each point may be displayed in a different color depending on its score. Alternatively, the point cloud may be converted into a mesh, and the score may be displayed as the color of its faces, vertices, edges, etc. Alternatively, the score may be displayed as a numerical value.

[0073] In other words, in estimating the attitude of the camera corresponding to a captured image, it is possible to determine for each captured image whether or not to perform bundle adjustment, and for captured images for which it is determined that bundle adjustment should be performed, pose estimation is performed using bundle adjustment and camera attitude information and point cloud data are derived, and for captured images for which it is determined that bundle adjustment should not be performed, pose estimation is performed without using bundle adjustment and camera attitude information is derived, the point cloud data is evaluated using the derived camera attitude information, and the display of the evaluation results is controlled.

[0074] Note that any method may be used to determine whether or not to perform global optimization (BA) using bundle adjustment. For example, the same method as used in the setting of transmission priorities described above may be used. For example, if the number of newly matching feature points is greater than a predetermined criterion (or equal to or greater than the predetermined criterion), global optimization using bundle adjustment may be performed in estimating the pose of the camera corresponding to the captured image. Furthermore, if the number of newly matching feature points is less than or equal to the predetermined criterion (or less than the predetermined criterion), global optimization using bundle adjustment may be omitted in estimating the pose of the camera corresponding to the captured image.

[0075] In the above description, the execution of such bundle adjustment is controlled by the above-described transmission control when attitude estimation is performed in the terminal device 61. However, the execution of bundle adjustment may be controlled in a case where attitude estimation is always performed in the terminal device 61 without performing the above-described transmission control.

[0076] Furthermore, even when pose estimation is performed on the server 62, global optimization (BA) using bundle adjustment may be omitted for some images, or a portion of it, in the pose estimation performed on the server 62, as in the case of the terminal device 61 described above. That is, whether or not to perform bundle adjustment in pose estimation of the pose of the camera corresponding to a captured image may be determined for each captured image. Then, for captured images for which it is determined that bundle adjustment should be performed, pose estimation may be performed using bundle adjustment to derive camera pose information and point cloud data. For captured images for which it is determined that bundle adjustment should not be performed, pose estimation may be performed without using bundle adjustment to derive camera pose information, and the point cloud data may be evaluated using the derived camera pose information. However, in this case, the point cloud evaluation results may be displayed on a device other than the server 62 (e.g., the terminal device 61). In this case, the server 62 may generate evaluation information indicating the point cloud evaluation results and provide it to the device.

[0077] As with the terminal device 61, the server 62 may use any method for determining whether to perform global optimization (BA) using bundle adjustment. For example, the same method as that used for setting the transmission priority described above may be used. That is, whether to perform bundle adjustment may be controlled based on the number of newly matching feature points in a captured image. For example, if the number of newly matching feature points is greater than (or equal to or greater than) a predetermined criterion, global optimization using bundle adjustment may be performed in estimating the pose of the camera corresponding to the captured image. Furthermore, if the number of newly matching feature points is less than (or equal to) a predetermined criterion, global optimization using bundle adjustment may be omitted in estimating the pose of the camera corresponding to the captured image.

[0078] Controlling transmission as described above prevents unnecessary increases in the amount of data transmitted, enabling faster pose estimation. Figure 11 shows an example of the amount of data transmitted. For example, assume that 900 original captured images encoded using the JPEG format result in a data volume of 8.2 GB. Reducing these captured images (JPEG images) to full HD size results in a data volume of 3.13 GB. Converting these captured images (JPEG images) to the HEIF format reduces the data volume to 1.57 GB, approximately half the size of a JPEG image. Extracting SIFT feature points from these captured images (HEIF images) with a maximum of 2048 feature points results in a data size of 353 MB. Extracting AKAZE feature points from these captured images (HEIF images) with a maximum of 2048 feature points results in a data size of 177 MB. In other words, the data size of AKAZE feature points is approximately half the size of SIFT feature points. Furthermore, the data size of AKAZE feature points is approximately one-tenth the size of a HEIF image. When the AKAZE features are encoded, the data size becomes even smaller, at 100 MB. By reducing the captured image, encoding it, extracting feature points, and selecting feature values ​​in this way, we can suppress the increase in the data volume of transmitted information. This allows for earlier posture estimation.

[0079] <3. First Embodiment> <3D Synthesis System> Fig. 12 is a block diagram showing an example of the configuration of a 3D synthesis system, which is one aspect of an information processing system to which the present technology is applied. The 3D synthesis system 100 shown in Fig. 12 is a system that captures an object, performs 3D modeling (e.g., photogrammetry) using the captured image to generate a 3D model (3D data) of the object, and displays a rendering image of the 3D model. The 3D synthesis system 100 may also control imaging based on the rendering image (3D model). For example, the present technology as described above in <2. Transmission Control> may be applied to the 3D synthesis system 100 (or each device thereof).

[0080] Note that Fig. 12 shows the main processing units, data flows, etc., and is not necessarily all that is shown in Fig. 12. In other words, the 3D composition system 100 may have devices and processing units that are not shown as blocks in Fig. 12. Also, there may be data flows and processes that are not shown as arrows, etc. in Fig. 12.

[0081] As shown in FIG. 12 , the 3D composition system 100 includes an imaging device 111, a terminal device 112, a server 113, an information processing device 114, and a display control device 115. The imaging device 111 is a device that captures an image of an object and generates a captured image. The imaging device 111 supplies the generated captured image to the terminal device 112. The imaging device 111 may be configured in any manner as long as it has an imaging function. For example, the imaging device 111 may be configured as a portable information processing device with an imaging function, such as a digital camera, a smartphone, a mobile phone with an imaging function, a tablet PC (Personal Computer) with an imaging function, or a mobile PC with an imaging function. The imaging device 111 may also have a communication function with the terminal device 112. The imaging device 111 may also be a system configured by multiple devices.

[0082] The imaging device 111 may further have an imaging control function, which may be used to change the camera attitude (internal parameters) and capture images. That is, the imaging range and other parameters related to imaging may be controlled by a user (photographer), and the shutter timing (the timing of generating a captured image) may be controlled by the imaging control function provided in the imaging device 111.

[0083] The imaging device 111 may further have a movement control function, and the movement control function may be used to change the camera attitude (external parameters). That is, the position, orientation, etc. of the imaging device 111 may be controlled by a user (photographer), or these may be controlled by a movement control function provided in the imaging device 111. That is, the imaging device 111 may be a mobile object such as a drone. Any method of movement using this movement control function may be used. For example, the imaging device 111 may be an aircraft with a flight function, a vehicle with a traveling function, a ship with a navigation function, or any other mobile object with a movement function.

[0084] Note that these imaging controls and movement controls may be performed based on instructions from the display control device 115. That is, the imaging and movement (i.e., the camera attitude) of the imaging device 111 may be controlled by the display control device 115. In this case, the imaging device 111 may have a communication function with the display control device 115.

[0085] The terminal device 112 may correspond to the terminal device 61 described above. That is, the present technology described above may be applied to the terminal device 112 in the same manner as in the case of the terminal device 61. For example, the terminal device 112 may generate transmission information using a captured image provided by the imaging device 111 and provide the transmission information to the server 113. For example, the terminal device 112 may use the captured image as the transmission information. Furthermore, the terminal device 112 may generate feature information from the captured image and provide the feature information as the transmission information. Furthermore, the terminal device 112 may perform posture estimation using the captured image to derive camera posture information. Furthermore, the terminal device 112 may provide the transmission information to the information processing device 114. The terminal device 112 may be configured in any manner as long as it has communication functions and information processing functions. For example, the terminal device 112 may be configured as a portable information processing device such as a smartphone, a mobile phone with an imaging function, a tablet PC, or a mobile PC.

[0086] The server 113 may correspond to the server 62 described above. That is, the present technology described above may be applied to the server 113 in the same manner as in the case of the server 62. For example, the server 113 may perform 3D modeling using transmission information (captured images or feature amount information) supplied from the terminal device 112. The 3D modeling method may be any method that uses the transmitted captured images or feature amount information. For example, the 3D modeling method may be the above-described photogrammetry. The following description will be given taking the case where photogrammetry is applied as an example. The server 113 may supply the 3D modeling results (e.g., a rendering image of a 3D model, etc.) to the display control device 115.

[0087] The server 113 may have any configuration as long as it can provide a 3D modeling function. For example, the server 113 may be a personal computer owned by a user (photographer), or a server owned by a company that provides 3D modeling services. The server 113 may be composed of a single information processing device, or a system composed of multiple information processing devices. For example, the server 113 may be an entity without a specified physical configuration, such as so-called cloud computing (cloud service).

[0088] The information processing device 114 may correspond to the above-described information processing device 63. That is, the above-described present technology may be applied to the information processing device 114 in the same manner as in the case of the information processing device 63. For example, the information processing device 114 may perform posture estimation using transmission information (captured image or feature amount information) supplied from the terminal device 112.

[0089] The display control device 115 is a device that displays the 3D modeling results supplied from the server 113. Furthermore, the display control device 115 may control the camera posture and imaging of the imaging device 111 based on the 3D modeling results. The display control device 115 may have any configuration as long as it has a display function. For example, it may be configured as a portable information processing device such as a smartphone, tablet PC, or mobile PC, or may be configured as a stationary information processing device (e.g., a personal computer). Furthermore, the display control device 115 may further have a communication function with the server 113. Furthermore, when controlling the imaging device 111, the display control device 115 may further have a communication function with the imaging device 111.

[0090] The display control device 115 may also be a system composed of multiple devices. For example, the display control device 115 may be composed of multiple information processing devices that perform processing in cooperation with each other. The display control device 115 may also be composed of a communication device that communicates with the server 113 and an information processing device that displays the 3D modeling results provided from the server 113 through that communication, accepts instructions from the user based on the display, and controls the imaging device 111.

[0091] The exchange of information (e.g., transmission information, 3D modeling results, etc.) between the imaging device 111, the terminal device 112, the server 113, the information processing device 114, and the display control device 115 may be performed via communication. For example, the imaging device 111, the terminal device 112, the server 113, the information processing device 114, and the display control device 115 may be communicably connected to each other via a network. This network may be any communication path, such as a wired communication path, a wireless communication path, or both. For example, it may be the Internet, a LAN, a WAN, Wi-Fi, etc. This network may also be a communication path such as USB or HDMI. This network may also include information processing devices such as a server or a relay device. This network may also be composed of multiple networks.

[0092] Furthermore, information may be exchanged between the terminal device 112 and the information processing device 114 via a recording medium (removable medium). This recording medium may be any type, such as a magnetic recording medium, an optical recording medium, a magneto-optical recording medium, a semiconductor memory, or any other recording medium.

[0093] <Terminal Device> Fig. 13 is a block diagram showing an example of the main configuration of the terminal device 112. Note that Fig. 13 shows the main processing units, data flows, etc., and does not necessarily show everything. In other words, the terminal device 112 may have processing units that are not shown as blocks in Fig. 13. Furthermore, there may be data flows or processes that are not shown as arrows, etc. in Fig. 13.

[0094] As shown in FIG. 13, the terminal device 112 has a control unit 131, a captured image acquisition unit 141, a transmission information generation unit 142, a transmission information supply unit 143, a bandwidth measurement unit 144, an attitude estimation unit 145, an output unit 146, an input unit 147, and a control information acquisition unit 148.

[0095] The control unit 131 controls each processing unit in the terminal device 112. For example, the control unit 131 controls the transmission information generation unit 142, the transmission information supply unit 143, and the attitude estimation unit 145. For example, the control unit 131 determines whether to supply a captured image or feature amount information generated from the captured image to a device that derives attitude information indicating the attitude of a camera corresponding to the captured image. Therefore, the control unit 131 can also be said to be a determination unit.

[0096] The control unit 131 may transmit only either the captured image or the feature amount information for each captured image. In this case, when it is determined that the captured image is to be transmitted, the control unit 131 may perform control so that the feature amount information corresponding to the captured image is not transmitted.

[0097] Furthermore, the control unit 131 may transmit both the captured image and the feature amount information for each captured image. For example, the control unit 131 may transmit a captured image whose feature amount information has been transmitted first later (when it is determined that the captured image is to be transmitted). Furthermore, when it is determined that the captured image is to be transmitted, the control unit 131 may transmit the feature amount information obtained from the captured image together with the captured image.

[0098] For example, the control unit 131 may determine whether to supply a captured image or feature amount information based on the use of the posture information. For example, as shown in the fourth row from the top of the table in Fig. 7 , when the use of the posture information is inspection, the control unit 131 may determine to supply a captured image as transmission information to the server 113. Furthermore, as shown in the fifth row from the top of the table in Fig. 7 , when the use of the posture information is content generation, the control unit 131 may determine to supply feature amount information as transmission information to the server 113.

[0099] The captured image acquisition unit 141 acquires the captured image supplied from the imaging device 111. The captured image acquisition unit 141 supplies the captured image to the transmission information generation unit 142.

[0100] The transmission information generation unit 142 generates transmission information using the captured image supplied thereto. The transmission information generation unit 142 has a captured image processing unit 151 and a feature information generation unit 152. The captured image processing unit 151 performs predetermined processing on the captured image. The captured image processing unit 151 supplies the processed captured image to the transmission information supply unit 143 as transmission information. The feature information generation unit 152 generates feature information using the captured image supplied thereto. The feature information generation unit 152 supplies the generated feature information as transmission information to the transmission information supply unit 143 and the posture estimation unit 145. Note that the transmission information generation unit 142 (captured image processing unit 151 and feature information generation unit 152) executes processing under the control of the control unit 131.

[0101] The transmission information supply unit 143 acquires the supplied transmission information (captured image or feature amount information) and supplies (transmits) it to the server 113. Therefore, the transmission information supply unit 143 can also be called a supply unit. The transmission information supply unit 143 executes processing under the control of the control unit 131.

[0102] The bandwidth measurement unit 144 measures the bandwidth of the transmission path (communication path) of the feature information. In other words, the bandwidth measurement unit 144 can also be referred to as a measurement unit. Note that any method for measuring the bandwidth may be used. The bandwidth measurement unit 144 supplies the measurement result (information indicating the measured bandwidth) to the control unit 131. The control unit 131 may determine whether to supply the captured image or the feature information based on the measurement result (measured bandwidth). For example, as shown in the first row from the top of the table in FIG. 7 , when the transmission path has a bandwidth wider than a predetermined standard, the control unit 131 may determine to supply the captured image to the server 113 as transmission information. Furthermore, as shown in the second row from the top of the table in FIG. 7 , when the transmission path has a bandwidth narrower than a predetermined standard, the control unit 131 may determine to supply the feature information to the server 62 as transmission information. Furthermore, as shown in the third row from the top of the table in FIG. 7 , when the transmission path has no bandwidth, the control unit 131 may determine to have the posture estimation unit 145 perform the remaining processing without transmitting the image information and feature information to the server 113. That is, in this case, camera posture information is derived in the posture estimation unit 145. Note that this no-bandwidth may include not only a non-connection state, but also an extremely narrow-bandwidth state in which feature amount information cannot be transmitted substantially.

[0103] Alternatively, the bandwidth measurement unit 144 may measure a delay time instead of the bandwidth of the transmission path and determine whether to supply the captured image or the feature information based on the delay time. For example, if the delay time is smaller than a predetermined first criterion, the control unit 131 may determine to supply the captured image to the server 113 as transmission information. Alternatively, if the delay time is medium (equal to or greater than the predetermined first criterion and smaller than a predetermined second criterion that is greater than the first criterion), the control unit 131 may determine to supply the feature information to the server 113 as transmission information. Alternatively, if the delay time is equal to or greater than the second criterion, the control unit 131 may determine to not transmit the image information and the feature information to the server 113 and to perform the remaining processing in the posture estimation unit 145. Note that this delay time includes the delay time due to the transmission of the transmission information. Furthermore, this delay time may further include the delay time (processing time) due to posture estimation performed by the server 113.

[0104] The posture estimation unit 145 uses the feature information supplied from the feature information generation unit 152 to execute the processes required to derive camera posture information that have not been executed by the feature information generation unit 152 (the remaining posture estimation processes). The posture estimation unit 145 evaluates the point cloud using the derived camera posture information. The posture estimation unit 145 then generates a display image showing the evaluation result, supplies it to the output unit 146, and causes it to be displayed.

[0105] The output unit 146 has a display device and displays a GUI, a display image, etc. For example, the output unit 146 displays a display image supplied from the posture estimation unit 145. The output unit 146 may also display a GUI for receiving user designations. For example, the output unit 146 may display the input screen 71 or the input screen 72 in FIG. 8 .

[0106] The input unit 147 has an input device and accepts user instructions input by the user operating the input device. Therefore, the input unit 147 can also be considered an instruction accepting unit. The input unit 147 supplies the accepted user instructions to the control unit 131. The control unit 131 may determine whether to supply a captured image or feature information based on the user instruction. For example, as shown in the sixth row from the top of the table in FIG. 7 , when a user instruction to supply a captured image is accepted, the control unit 131 may determine to supply the captured image to the server 113 as transmission information in accordance with the user instruction. Furthermore, as shown in the seventh row from the top of the table in FIG. 7 , when a user instruction to supply feature information is accepted, the control unit 131 may determine to supply the feature information to the server 113 as transmission information in accordance with the user instruction. Furthermore, as shown in the eighth row from the top of the table in FIG. 7 , when a user instruction to not transmit transmission information to the server 113 is accepted, the control unit 131 may determine to have the posture estimation unit 145 perform posture estimation in accordance with the user instruction.

[0107] The control information acquisition unit 148 acquires control information, supplied from the server 113, for controlling whether to supply a captured image or feature information, and supplies the control information to the control unit 131. The control unit 131 may determine whether to supply a captured image or feature information based on the control information. For example, as shown in the ninth row from the top of the table in FIG. 7 , when control information instructing the server 113 to supply a captured image is acquired from the server 113, the control unit 131 may determine to supply the captured image to the server 113 as transmission information in accordance with the control information. Furthermore, as shown in the tenth row from the top of the table in FIG. 7 , when control information instructing the server 113 to supply feature information is acquired from the server 113, the control unit 131 may determine to supply the feature information to the server 113 as transmission information in accordance with the control information. Furthermore, as shown in the eleventh row from the top of the table in FIG. 7 , when control information instructing the server 113 not to transmit transmission information to the server 113 is acquired from the server 113, the control unit 131 may determine to cause the posture estimation unit 145 to perform posture estimation in accordance with the control information.

[0108] Note that the control unit 131 may cause the information processing device 114 to derive the camera posture information instead of causing the posture estimation unit 145 to derive the camera posture information. In other words, the control unit 131 may supply the captured image or feature amount information to the information processing device 114, and cause the information processing device 114 to derive the camera posture information.

[0109] Furthermore, the control unit 131 may control whether posture estimation is performed in the posture estimation unit 145 or in the information processing device 114. Furthermore, when posture estimation is performed in the information processing device 114, the control unit 131 may also control transmission information for this information processing device 114 in the same way as for transmission information for the server 113. In other words, the control unit 131 may control whether to supply a captured image or feature amount information.

[0110] <Transmission Information Generator> Fig. 14 is a block diagram showing an example of the main configuration of the transmission information generator 142. Note that Fig. 14 shows the main processing units, data flows, etc., and is not limited to all that is shown in Fig. 14. In other words, the transmission information generator 142 may have processing units that are not shown as blocks in Fig. 14. Furthermore, there may be data flows or processes that are not shown as arrows or the like in Fig. 14.

[0111] 14 , the captured image processing unit 151 of the transmission information generation unit 142 has a captured image reduction unit 1661, a captured image selection unit 162, a captured image encoding unit 163, and a priority setting unit 164. The feature information generation unit 152 of the transmission information generation unit 142 has an ROI setting unit 171, a feature point extraction unit 172, a feature point matching unit 173, a posture estimation unit 174, and a feature amount encoding unit 175.

[0112] The captured image reducing unit 161 reduces the size (resolution) of the captured image supplied from the captured image acquiring unit 141. The size (resolution) of the captured image supplied from the captured image acquiring unit 141 may be any size (resolution). Furthermore, the size (resolution) of the captured image after reduction may be any size (resolution). For example, a captured image of 4K size (3840 x 2160 pixels) may be reduced to full HD size (1920 x 1080 pixels). The captured image reducing unit 161 supplies the reduced captured image to the captured image selecting unit 162.

[0113] The captured image selection unit 162 selects a captured image to be supplied to the server 113 from the supplied captured images. For example, the captured image selection unit 162 may select a captured image to be supplied to the server 113 based on the quality of the captured image. For example, the captured image selection unit 162 may control so that a captured image of lower quality than a predetermined standard is not supplied to the server 113. For example, the captured image selection unit 162 may acquire feature points extracted from the captured image by the feature point extraction unit 172, and may determine a captured image having fewer extracted feature points than a predetermined standard as a low-quality captured image and not supply the image to the server 113. In other words, the captured image selection unit 162 may select a captured image of higher quality than the predetermined standard as a captured image to be supplied to the server 113. The captured image selection unit 162 supplies the selected captured image to the captured image encoding unit 163. The captured image is supplied to the transmission information supply unit 143 via the captured image encoding unit 163 and the priority setting unit 164. That is, when it is determined that the captured image is to be supplied, the supply unit supplies the captured image selected by the captured image selection unit 162 to the server 113 .

[0114] The captured image encoding unit 163 encodes (compresses) the captured images supplied from the captured image selection unit 162. Any encoding method may be used. For example, the captured image encoding unit 163 may encode the captured images in the JPEG format. Alternatively, the captured image encoding unit 163 may encode the captured images in the HEIF format. The captured image encoding unit 163 supplies the encoded captured images to the priority setting unit 164.

[0115] The priority setting unit 164 sets a priority for supplying captured images (encoded data) supplied from the captured image encoding unit 163. For example, when captured images are supplied as transmission information, the priority setting unit 164 sets a priority for supplying the captured images. Any method for setting this priority may be used. For example, the priority setting unit 164 may acquire a feature point matching result from the feature point matching unit 173 and set a higher priority for a captured image with more newly matching feature points. The priority setting unit 164 supplies information indicating the priority together with the captured image to the transmission information supply unit 143. The transmission information supply unit 143 supplies the captured image according to the priority.

[0116] The ROI setting unit 171 sets a region of interest from which feature points are extracted for the captured image supplied from the captured image acquisition unit 141. The ROI setting unit 171 supplies information indicating the set region of interest to the feature point extraction unit 172. In this case, the feature point extraction unit 172 may extract feature points from the region of interest set by the ROI setting unit 171.

[0117] The feature point extraction unit 172 extracts feature points from the captured image supplied from the captured image acquisition unit 141, similar to the example described with reference to FIG. 2 . For example, the feature points may be included in feature amount information. That is, the feature point extraction unit 172 may supply the extracted feature points to the feature amount encoding unit 175 as feature amount information. This feature amount information is supplied to the transmission information supply unit 143 via the feature amount encoding unit 175. That is, when it is decided to supply feature amount information, the transmission information supply unit 143 may supply the feature points extracted from the captured image by the feature point extraction unit 172 to the server 113 as feature amount information (transmission information).

[0118] The control unit 131 may further determine feature amounts of the captured image to be supplied as feature amount information, and the feature point extraction unit 172 may extract feature points corresponding to the determined feature amounts. For example, the control unit 131 may determine feature amounts to be applied from among candidate methods such as SIFT, AKAZE, and ORB. For example, the control unit 131 may determine feature amounts to be applied based on the bandwidth (or delay time) of the transmission path. The control unit 131 may also determine feature amounts to be applied based on a use case (application of posture information). The control unit 131 may also determine feature amounts to be applied based on manual selection (e.g., user instruction). The control unit 131 may also determine feature amounts to be applied based on control information.

[0119] The feature point extraction unit 172 may also supply the extracted feature points to the captured image selection unit 162. The feature point extraction unit 172 may also supply the extracted feature points to the feature point matching unit 173.

[0120] Similar to the example described with reference to FIG. 2 , the feature point matching unit 173 matches feature points supplied from the feature point extraction unit 172 between captured images. For example, the matching result (feature point matching result) may be included in the feature amount information. That is, the feature point matching unit 173 may supply the feature point matching result to the feature amount encoding unit 175 as feature amount information. This feature amount information is supplied to the transmission information supply unit 143 via the feature amount encoding unit 175. That is, when it is determined to supply the feature amount information, the transmission information supply unit 143 may supply the feature point matching result obtained by the feature point matching unit 173 to the server 113 as feature amount information (transmission information). Furthermore, the feature point matching unit 173 may supply the feature point matching result to the priority setting unit 164. Furthermore, the feature point matching unit 173 may supply the feature point matching result to the posture estimation unit 174.

[0121] As in the example described with reference to FIG. 2 , posture estimation unit 174 uses the feature point matching result to perform posture estimation up to intermediate processing not including bundle adjustment, and generates intermediate data (also referred to as posture estimation intermediate data). For example, the posture estimation intermediate data may be included in feature information. That is, posture estimation unit 174 may supply the posture estimation intermediate data to feature encoding unit 175 as feature information. This feature information is supplied to transmission information supply unit 143 via feature encoding unit 175. That is, when it is determined to supply feature information, transmission information supply unit 143 may supply the posture estimation intermediate data derived in posture estimation unit 174 to server 113 as feature information (transmission information).

[0122] The feature amount encoding unit 175 acquires feature points supplied from the feature point extraction unit 172. The feature amount encoding unit 175 also acquires feature point matching results supplied from the feature point matching unit 173. The feature amount encoding unit 175 also acquires posture estimation intermediate data supplied from the posture estimation unit 174. The feature amount encoding unit 175 encodes the supplied feature amount information (feature points, feature point matching results, posture estimation intermediate data, etc.). Any encoding method may be used. The feature amount encoding unit 175 supplies the encoded data of the feature amount information to the transmission information supply unit 143 as transmission information.

[0123] <Posture Estimation Unit> Fig. 15 is a block diagram showing an example of the main configuration of posture estimation unit 145. Note that Fig. 15 shows the main processing units, data flows, etc., and does not necessarily show everything. In other words, posture estimation unit 145 may have processing units that are not shown as blocks in Fig. 15. Furthermore, there may be data flows or processes that are not shown as arrows or the like in Fig. 15.

[0124] As shown in FIG. 15 , the posture estimation unit 145 has a feature point matching unit 181 , a BA execution determination unit 182 , a posture estimation unit 183 , a posture estimation unit 184 , a point cloud evaluation unit 185 , and an evaluation result display control unit 186 .

[0125] The feature point matching unit 181 acquires feature amount information supplied from the feature amount information generation unit 152. If this feature amount information includes feature points extracted by the feature point extraction unit 172, the feature point matching unit 181 matches the feature points between captured images. The feature point matching unit 181 supplies the obtained feature point matching results to the BA execution determination unit 182. Note that if the feature amount information includes feature point matching results or intermediate posture estimation data (if feature point matching has already been performed), the feature point matching unit 181 omits feature point matching and supplies the obtained feature amount information to the BA execution determination unit 182.

[0126] The BA execution determination unit 182 determines whether to perform global optimization (BA) using bundle adjustment in pose estimation for the captured image to be processed. Therefore, the BA execution determination unit 182 can also be considered a determination unit. Note that any method for determining whether to perform global optimization (BA) using bundle adjustment may be used. For example, the same method as used in the transmission priority setting described above may be used. For example, if the number of newly matching feature points is greater than or equal to a predetermined criterion, the BA execution determination unit 182 may determine to perform global optimization using bundle adjustment in pose estimation of the camera corresponding to the captured image. Alternatively, if the number of newly matching feature points is less than or equal to a predetermined criterion, the BA execution determination unit 182 may determine to omit global optimization using bundle adjustment in pose estimation of the camera corresponding to the captured image. If it is determined that global optimization using bundle adjustment is to be performed, the BA execution determination unit 182 supplies feature amount information of the captured image to the pose estimation unit 183. Furthermore, if it is determined that global optimization by bundle adjustment is not to be performed, the BA execution determination unit 182 supplies feature amount information of the captured image to the posture estimation unit 184 .

[0127] The posture estimation unit 183 performs posture estimation, which executes global optimization by bundle adjustment, using the feature amount information supplied from the BA execution determination unit 182. Through this posture estimation, the posture estimation unit 183 generates highly accurate camera posture information and a highly accurate sparse point cloud. Therefore, the posture estimation unit 183 can also be said to be a first posture estimation unit. The posture estimation unit 183 supplies the generated information to the point cloud evaluation unit 185.

[0128] The posture estimation unit 184 uses the feature amount information supplied from the BA execution determination unit 182 to perform posture estimation without performing global optimization using bundle adjustment. Through this posture estimation, the posture estimation unit 184 generates low-accuracy camera posture information. Therefore, the posture estimation unit 184 can also be considered a second posture estimation unit. The posture estimation unit 184 supplies the generated camera posture information to the point cloud evaluation unit 185.

[0129] The point cloud evaluation unit 185 uses the camera posture information supplied from the posture estimation unit 183 and the posture estimation unit 184 to evaluate the sparse point cloud (high accuracy) supplied from the posture estimation unit 183 and derive an evaluation value (score). Therefore, the point cloud evaluation unit 185 can also be said to be an evaluation unit that evaluates point cloud data using posture information. The point cloud evaluation unit 185 supplies the score to the evaluation result display control unit 186.

[0130] The evaluation result display control unit 186 generates a display image that displays the score of each point in the point cloud supplied from the point cloud evaluation unit 185 as the evaluation result, and outputs (displays) it to the output unit 146. In other words, the evaluation result display control unit 186 can also be said to be a display control unit that controls the display of the results of the point cloud evaluation.

[0131] The above-described configuration allows the terminal device 112 to apply the present technology described above in <2. Transmission Control>, thereby enabling earlier posture estimation.

[0132] <Server> Fig. 16 is a block diagram showing an example of the main configuration of the server 113. Note that Fig. 16 shows the main processing units, data flows, etc., and is not limited to all that is shown in Fig. 16. In other words, the server 113 may have processing units that are not shown as blocks in Fig. 16. Furthermore, there may be data flows or processes that are not shown as arrows or the like in Fig. 16.

[0133] 16 , the server 113 includes a control unit 201, an acquisition unit 211, a captured image decoding unit 212, a feature point extraction unit 213, a feature point matching unit 214, a posture estimation unit 215, a feature amount information decoding unit 216, a storage unit 217, a depth image estimation unit 218, a fusion processing unit 219, and a supply unit 220. The server 113 also includes a bandwidth measurement unit 221, a control information generation unit 222, and a control information supply unit 223.

[0134] The control unit 201 controls each processing unit (acquisition unit 211 to control information supply unit 223) in the server 113.

[0135] The acquisition unit 211 acquires a captured image or feature information (feature information generated from the captured image) supplied as transmission information from the terminal device 112. When the acquired captured image is acquired, the acquisition unit 211 supplies the captured image to the captured image decoding unit 212. When the acquired feature information is acquired, the acquisition unit 211 supplies the feature information to the feature information decoding unit 216. That is, the acquisition unit 211 controls the processing to be executed according to the supplied transmission information. That is, the acquisition unit 211 executes the remaining processing for deriving posture information that has not been executed by the terminal device 61 according to the acquired transmission information. The control of which processing to execute may be performed based on any information. For example, flag information or the like indicating what information will be transmitted as transmission information may be supplied from the terminal device 112, and the acquisition unit 211 may determine what information the transmission information is based on the flag information and select whether to supply the transmission information to the captured image decoding unit 212 or the feature information decoding unit 216. Furthermore, the acquisition unit 211 may ascertain what kind of information the transmission information is based on the data amount of the transmission information, and select whether to supply the information to the captured image decoding unit 212 or the feature amount information decoding unit 216. Furthermore, the acquisition unit 211 may ascertain what kind of information the transmission information is based on the extension of the transmission information, and select whether to supply the information to the captured image decoding unit 212 or the feature amount information decoding unit 216. Of course, the acquisition unit 211 may also select whether to supply the information to the captured image decoding unit 212 or the feature amount information decoding unit 216 based on other information.

[0136] The captured image decoding unit 212 decodes the supplied captured image and supplies it to the feature point extraction unit 213. Similar to the example described with reference to FIG. 2, the feature point extraction unit 213 extracts feature points from the captured image and supplies the feature points and the captured image to the feature point matching unit 214. Similar to the example described with reference to FIG. 2, the feature point matching unit 214 matches feature points between the captured images and supplies the feature point matching results and the captured image to the posture estimation unit 215. Similar to the example described with reference to FIG. 2, the posture estimation unit 215 estimates the posture of the camera corresponding to the captured image based on the feature point matching results and derives camera posture information and a sparse point cloud. The posture estimation unit 215 supplies the derived camera posture information and sparse point cloud together with the captured image to the storage unit 217 for storage. Furthermore, the posture estimation unit 215 may supply the derived camera posture information and sparse point cloud together with the captured image to the depth image estimation unit 218.

[0137] The feature amount information decoding unit 216 decodes the supplied feature amount information. For example, if the feature amount information includes feature points, the feature amount information decoding unit 216 supplies the feature points to the feature point matching unit 214. The feature point matching unit 214 matches feature points between captured images, similar to the example described with reference to FIG. 2 , and supplies the feature point matching result to the posture estimation unit 215.

[0138] Furthermore, if the feature amount information includes a feature point matching result or intermediate posture estimation data, the feature amount information decoding unit 216 supplies the feature point matching result or intermediate posture estimation data to the posture estimation unit 215. Similar to the example described with reference to Fig. 2, the posture estimation unit 215 estimates the posture of the camera corresponding to the captured image based on the feature point matching result or intermediate posture estimation data, and derives camera posture information and a sparse point cloud.

[0139] That is, the feature information decoding unit 216 of the captured image decoding unit 212 executes processing for deriving posture information in accordance with the supplied transmission information. That is, the feature information decoding unit 216 of the captured image decoding unit 212 executes the remaining processing for deriving posture information that has not been executed in the terminal device 102 in accordance with the acquired transmission information. That is, the feature information decoding unit 216 of the captured image decoding unit 212 can also be considered a posture information derivation unit that derives posture information indicating the posture of the camera corresponding to the captured image using the acquired captured image or feature information. Note that the feature information decoding unit 216 of the captured image decoding unit 212 may sequentially derive derivable posture information using the acquired captured image or feature information. Alternatively, the feature information decoding unit 216 of the captured image decoding unit 212 may acquire all captured images or feature information and then derive all posture information at once.

[0140] Furthermore, the captured image decoding unit 212 and the feature amount information decoding unit 216 may set priorities for deriving posture information and derive posture information according to the priorities. Any method for setting the priorities may be used. For example, the same method as the above-described method for setting the transmission priorities may be used. For example, as described above, a higher priority may be set for a captured image with a larger number of newly matching feature points.

[0141] The storage unit 217 has a storage medium and stores the camera posture information, sparse point cloud, and captured images supplied from the posture estimation unit 215. The storage unit 217 may also supply this information to the depth image estimation unit 218. The storage unit 217 may also acquire and store 3D data supplied from the fusion processing unit 219. The storage unit 217 may also supply the stored information (e.g., 3D data, camera posture information, sparse point cloud, captured images, etc.) to the supply unit 220.

[0142] The depth image estimation unit 218 acquires the camera posture information, sparse point cloud, and captured image supplied from the posture estimation unit 215. The depth image estimation unit 218 may also read this information from the storage unit 217. Using this information, the depth image estimation unit 218 estimates a depth image by stereo matching, similar to the example described with reference to FIG. 2. The depth image estimation unit 218 supplies the estimated depth image to the fusion processing unit 219. Similar to the example described with reference to FIG. 2, the fusion processing unit 219 fuses the supplied depth values ​​to reconstruct 3D data. The fusion processing unit 219 may supply the generated 3D data to the storage unit 217 for storage. The fusion processing unit 219 may also supply the generated 3D data to the supply unit 220.

[0143] The supply unit 220 has a communication function for communicating with other devices such as the display control device 115, and may read information (e.g., 3D data, camera attitude information, sparse point cloud, captured image, etc.) from the storage unit 217 and supply the information to the other devices. The supply unit 220 may also acquire 3D data supplied from the fusion processing unit 219 and supply the 3D data to the other devices.

[0144] The bandwidth measurement unit 221 measures the bandwidth of the transmission path (communication path) of the feature information. In other words, the bandwidth measurement unit 221 can also be called a measurement unit. Note that any method for measuring this bandwidth may be used. The bandwidth measurement unit 221 supplies the measurement result (information indicating the measured bandwidth) to the control information generation unit 222. Based on the measurement result, the control information generation unit 222 generates control information for controlling whether the terminal device 112 supplies the captured image to the server 113 or whether the terminal device 112 supplies the feature information to the server 113. The control information generation unit 222 supplies the control information to the control information supply unit 223. The control information supply unit 223 supplies the control information supplied from the control information generation unit 222 to the terminal device 112 that supplies the captured image or the feature information.

[0145] <Posture Estimation Unit> Fig. 17 is a block diagram showing an example of the main configuration of the posture estimation unit 215. Note that Fig. 17 shows the main processing units, data flows, etc., and does not necessarily show everything. In other words, the posture estimation unit 215 may have processing units that are not shown as blocks in Fig. 17. Furthermore, there may be data flows or processes that are not shown as arrows or the like in Fig. 17.

[0146] As shown in FIG. 17, the posture estimation unit 215 includes a BA execution determination unit 231 , a posture estimation unit 232 , a posture estimation unit 233 , a point cloud evaluation unit 234 , and an evaluation information generation unit 235 .

[0147] The BA execution determination unit 231 determines whether to perform global optimization (BA) using bundle adjustment in pose estimation for a captured image to be processed. Therefore, the BA execution determination unit 231 can also be considered a determination unit. Note that any method for determining whether to perform global optimization (BA) using bundle adjustment may be used. For example, the same method as used in the transmission priority setting described above may be used. For example, if the number of newly matching feature points is greater than or equal to a predetermined standard, the BA execution determination unit 231 may determine to perform global optimization using bundle adjustment in pose estimation of the camera corresponding to the captured image. Furthermore, if the number of newly matching feature points is less than or equal to a predetermined standard, the BA execution determination unit 231 may determine to omit global optimization using bundle adjustment in pose estimation of the camera corresponding to the captured image. If it is determined that global optimization using bundle adjustment is to be performed, the BA execution determination unit 231 supplies feature amount information of the captured image to the pose estimation unit 232. Furthermore, if it is determined that global optimization by bundle adjustment is not to be performed, the BA execution determination unit 231 supplies feature amount information of the captured image to the posture estimation unit 233 .

[0148] The posture estimation unit 232 performs posture estimation, which involves performing overall optimization by bundle adjustment, using the feature amount information supplied from the BA execution determination unit 231. Through this posture estimation, the posture estimation unit 232 generates highly accurate camera posture information and a highly accurate sparse point cloud. Therefore, the posture estimation unit 232 can also be considered a first posture estimation unit. The posture estimation unit 232 supplies the generated information to the point cloud evaluation unit 234.

[0149] The posture estimation unit 233 uses the feature amount information supplied from the BA execution determination unit 231 to perform posture estimation without performing global optimization using bundle adjustment. Through this posture estimation, the posture estimation unit 233 generates low-accuracy camera posture information. Therefore, the posture estimation unit 233 can also be considered a second posture estimation unit. The posture estimation unit 233 supplies the generated camera posture information to the point cloud evaluation unit 234.

[0150] The point cloud evaluation unit 234 uses the camera posture information supplied from the posture estimation unit 232 and the posture estimation unit 233 to evaluate the sparse point cloud (high precision) supplied from the posture estimation unit 232 and derive an evaluation value (score). Therefore, the point cloud evaluation unit 234 can also be said to be an evaluation unit that evaluates point cloud data using posture information. The point cloud evaluation unit 234 supplies the score to the evaluation information generation unit 235.

[0151] The evaluation information generation unit 235 generates evaluation information indicating the score (evaluation result) of each point in the point cloud supplied from the point cloud evaluation unit 234. The evaluation information generation unit 235 may supply the evaluation information to the storage unit 217 for storage. The evaluation information generation unit 235 may also supply the evaluation information to the supply unit 220 and supply it to another device.

[0152] The server 113 having the above configuration can apply the present technology described above in <2. Transmission Control>, thereby enabling posture estimation at an earlier stage.

[0153] <Flow of System Processing> The 3D composition system 100 having the devices configured as described above executes system processing, for example, according to the flow shown in the flowchart in Fig. 18. Note that Fig. 18 omits the illustration of the processing of the information processing device 114.

[0154] For example, the imaging device 111 captures an image of an object in step S111. The imaging device 111 supplies the captured image generated by the imaging to the terminal device 112. The terminal device 112 executes transmission control processing in step S121, generates transmission information using the captured image, and supplies the information to the server 113. The server 113 executes 3D modeling processing in step S131, and reconstructs 3D data using the transmission information. The server 113 generates display images and control information as needed, and supplies the information to the display control device 115. The display control device 115 acquires the information in step S141, and performs display and control in step S142.

[0155] <Flow of Transmission Control Process> An example of the flow of the transmission control process executed in step S121 of FIG. 18 will be described with reference to the flowcharts of FIGS.

[0156] In step S151, the captured image acquisition unit 141 acquires a captured image. In step S152, the control unit 131 executes a transmission information determination process to determine transmission information. In step S153, the captured image processing unit 151 determines whether or not to transmit the captured image. If it is determined that the captured image is to be transmitted, the process proceeds to step S154.

[0157] In step S154, the captured image reducing unit 161 reduces the size of the captured image. In step S155, the captured image selecting unit 162 selects a captured image to be transmitted. In step S156, the captured image encoding unit 163 encodes the captured image to be transmitted. In step S157, the priority setting unit 164 sets priorities. In step S158, the transmission information supplying unit 143 transmits the captured image to the server 113 in accordance with the priorities.

[0158] When the process of step S158 ends, the process proceeds to step S161 in Fig. 20. Also, if it is determined in step S153 in Fig. 19 that the captured image is not to be transmitted, the process proceeds to step S161 in Fig. 20.

[0159] In step S161 of FIG. 20 , the feature information generation unit 152 determines whether or not to transmit feature information. If it is determined that feature information is to be transmitted, the process proceeds to step S162. In step S162, the ROI setting unit 171 sets a region of interest (ROI). In step S163, the feature point extraction unit 172 extracts feature points from the ROI. Note that at this time, the feature point extraction unit 172 may determine feature amounts to apply and extract feature points of the determined feature amounts. In step S164, the feature point matching unit 173 matches feature points. In step S165, the posture estimation unit 174 performs posture estimation halfway and generates posture estimation intermediate data. In step S166, the feature encoding unit 175 encodes the feature information (feature points, feature point matching results, posture estimation intermediate data, etc.) generated as described above. In step S167 , the transmission information supplying unit 143 transmits the feature amount information to the server 113 .

[0160] When the process of step S167 ends, the process proceeds to step S168. If it is determined in step S161 that the feature amount information is not to be transmitted, the process proceeds to step S168.

[0161] In step S168, the control unit 131 determines whether or not to transmit the transmission information to the information processing device 114. If it is determined that the transmission information should be transmitted to the information processing device 114, the process proceeds to step S169. In step S169, the transmission information generation unit 142 processes the captured image or generates feature amount information and transmits it to the information processing device 114. When the process of step S169 ends, the process proceeds to step S181 in Fig. 21. Also, if it is determined in step S168 in Fig. 20 that the transmission information should not be transmitted to the information processing device, the process proceeds to step S181 in Fig. 21.

[0162] In step S181 of FIG. 21 , the control unit 131 determines whether or not to perform posture estimation in the terminal device 112. If it is determined that posture estimation is to be performed in the terminal device 112, the process proceeds to step S182. In step S182, the feature point matching unit 181 generates a feature point matching result or posture estimation intermediate data. In step S183, the BA execution determination unit 182 determines whether or not to perform bundle adjustment. If it is determined that bundle adjustment is to be performed, the process proceeds to step S184. In step S184, the posture estimation unit 183 performs posture estimation using bundle adjustment. When the process of step S184 ends, the process proceeds to step S186. Also, if it is determined in step S183 that bundle adjustment is not to be performed, the process proceeds to step S185. In step S185, the posture estimation unit 184 performs posture estimation without using bundle adjustment. When the process of step S185 ends, the process proceeds to step S186.

[0163] In step S186, the point cloud evaluation unit 185 evaluates the point cloud. In step S187, the evaluation result display control unit 186 displays the evaluation result. When the processing of step S187 ends, the processing proceeds to step S188. Also, if it is determined in step S181 that posture estimation will not be performed in the terminal device 112, the processing proceeds to step S188.

[0164] In step S188, the control unit 131 determines whether or not to end the transmission control process. If it is determined not to end the transmission control process, the process returns to step S151 in Fig. 19, and the subsequent processes are repeated. Also, if it is determined in step S188 in Fig. 21 that the transmission control process is to end, the transmission control process ends, and the process returns to Fig. 18.

[0165] <Flow of Transmission Information Determination Process> Next, an example of the flow of the transmission information determination process executed in step S152 of Fig. 19 will be described with reference to the flowchart of Fig. 22. In this example, the transmission information is determined based on the bandwidth of the transmission path.

[0166] When the transmission information determination process starts, in step S201, the bandwidth measurement unit 144 measures the bandwidth. In step S202, the control unit 131 determines whether the bandwidth of the transmission path is wideband based on the measurement result. If it is determined to be wideband, the process proceeds to step S203. In step S203, the control unit 131 determines to transmit the captured image. When the process of step S203 ends, the transmission information determination process ends, and the process returns to FIG. 19 .

[0167] 22, if it is determined that the bandwidth of the transmission path is not wideband, the process proceeds to step S204. In step S204, the control unit 131 determines whether the bandwidth of the transmission path is narrowband or not based on the measurement results. If it is determined that the bandwidth is narrowband, the process proceeds to step S205. In step S205, the control unit 131 determines to transmit feature information. When the process of step S205 ends, the transmission information determination process ends, and the process returns to FIG. 19.

[0168] 22, if it is determined that the bandwidth of the transmission path is not narrowband (no bandwidth), the process proceeds to step S206. In step S206, the control unit 131 determines the process to be performed by the terminal device 112 or the information processing device 114. When the process of step S206 ends, the transmission information determination process ends, and the process returns to FIG.

[0169] <Flow of Transmission Information Determination Process> Next, an example of the flow of the transmission information determination process executed in step S152 of Fig. 19 will be described with reference to the flowchart of Fig. 23. In this example, the transmission information is determined based on a use case (application of posture information).

[0170] When the transmission information determination process starts, in step S211, the control unit 131 determines whether the use of the posture information is for inspection. If it is determined that the use is for inspection, the process proceeds to step S212. In step S212, the control unit 131 determines to transmit the captured image. When the process of step S212 ends, the transmission information determination process ends, and the process returns to FIG. 19 .

[0171] 23, if it is determined that the purpose of the posture information is not inspection (content generation), the process proceeds to step S219. In step S219, the control unit 131 determines the transmission of feature information. When the process of step S213 ends, the transmission information determination process ends, and the process returns to FIG. 19.

[0172] <Flow of Transmission Information Determination Process> Next, an example of the flow of the transmission information determination process executed in step S152 of Fig. 19 will be described with reference to the flowchart of Fig. 24. In this example, the transmission information is determined based on a user instruction.

[0173] When the transmission information determination process starts, in step S221, the input unit 147 accepts a user instruction specifying whether to transmit a captured image or feature information as the transmission information. In step S222, the control unit 131 determines whether transmission of the captured image has been specified based on the user instruction. If it is determined that transmission of the captured image has been specified, the process proceeds to step S223. In step S223, the control unit 131 determines to transmit the captured image. When the process of step S223 ends, the transmission information determination process ends, and the process returns to FIG. 19 .

[0174] Furthermore, if it is determined in step S222 of Fig. 24 that the transmission of the captured image has not been designated, the process proceeds to step S224. In step S224, the control unit 131 determines whether or not the transmission of feature information has been designated based on the accepted user instruction. If it is determined that the transmission of feature information has been designated, the process proceeds to step S225. In step S225, the control unit 131 determines the transmission of the feature information. When the process of step S225 ends, the transmission information determination process ends, and the process returns to Fig. 19 .

[0175] 24, if it is determined that transmission of feature information is not specified (transmission information is not to be transmitted to the server 113), the process proceeds to step S226. In step S226, the control unit 131 determines the process to be performed in the terminal device 112 or the information processing device 114. When the process of step S226 ends, the transmission information determination process ends, and the process returns to FIG.

[0176] <Flow of Transmission Information Determination Process> Next, an example of the flow of the transmission information determination process executed in step S152 of Fig. 19 will be described with reference to the flowchart of Fig. 25. In this example, the transmission information is determined based on the control information.

[0177] When the transmission information determination process starts, in step S231, the control information acquisition unit 148 acquires control information that controls whether to transmit the captured image or the feature amount information as the transmission information. In step S232, the control unit 131 determines whether the transmission of the captured image has been designated based on the control information. If it is determined that the transmission of the captured image has been designated, the process proceeds to step S233. In step S233, the control unit 131 determines to transmit the captured image. When the process of step S233 ends, the transmission information determination process ends, and the process returns to FIG. 19 .

[0178] Furthermore, if it is determined in step S232 of Fig. 25 that the transmission of the captured image has not been designated, the process proceeds to step S234. In step S234, the control unit 131 determines whether or not the transmission of feature information has been designated based on the acquired control information. If it is determined that the transmission of feature information has been designated, the process proceeds to step S235. In step S235, the control unit 131 determines the transmission of the feature information. When the process of step S235 ends, the transmission information determination process ends, and the process returns to Fig. 19.

[0179] 25, if it is determined that transmission of feature information is not specified (transmission information is not to be transmitted to the server 113), the process proceeds to step S236. In step S236, the control unit 131 determines the process to be performed in the terminal device 112 or the information processing device 114. When the process of step S236 ends, the transmission information determination process ends, and the process returns to FIG.

[0180] By performing the processes as described above, the terminal device 112 can apply the present technology described above in <2. Transmission Control>. Therefore, the attitude can be estimated earlier.

[0181] <3D Modeling Process Flow> An example of the flow of the 3D modeling process executed in step S131 of FIG. 18 will be described with reference to the flowcharts of FIGS. 26 and 27.

[0182] When the 3D modeling process starts, in step S251, the acquisition unit 211 acquires transmission information. In step S252, the acquisition unit 211 determines whether or not a captured image has been acquired. If it is determined that a captured image has been acquired, the process proceeds to step S253.

[0183] In step S253, the captured image decoding unit 212 decodes the captured image. In step S254, the feature point extraction unit 213 extracts feature points. In step S255, the feature point matching unit 214 performs feature point matching. In step S256, the posture estimation unit 215 estimates the posture.

[0184] In step S257, the storage unit 217 stores the camera posture information generated by posture estimation, the sparse point cloud, the captured image, etc. In step S258, the depth image estimation unit 218 estimates a depth image by stereo matching. In step S259, the fusion processing unit 219 fuses the depth values. In step S260, the storage unit 217 stores the generated 3D data. In step S261, the supply unit 220 supplies information such as the 3D data.

[0185] When the process of step S261 ends, the 3D modeling process ends, and the process returns to FIG.

[0186] Also, if it is determined in step S252 of FIG. 26 that a captured image has not been acquired (feature amount information has been acquired), the process proceeds to step S271 of FIG.

[0187] In step S271 of FIG. 27 , the feature amount information decoding unit 216 decodes the feature amount information. In step S272, the feature amount information decoding unit 216 determines whether feature points have been acquired as feature amount information. If it is determined that feature points have been acquired, the process proceeds to step S273. In step S273, the feature point matching unit 214 performs matching of the feature points. When the process of step S273 ends, the process proceeds to step S274. Also, if it is determined in step S272 that feature points have not been acquired as feature amount information (i.e., feature point matching results or posture estimation intermediate data have been acquired), the process proceeds to step S274. In step S274, the posture estimation unit 215 estimates posture using the feature point matching results or posture estimation intermediate data. When the process of step S274 ends, the process returns to step S257 of FIG. 26 .

[0188] <Flow of Transmission Control Processing> As described above, the server 113 can also generate control information for controlling transmission information and supply it to the terminal device 112. An example of the flow of the transmission control processing will be described with reference to the flowchart in FIG.

[0189] When the transmission control process starts, in step S281, the bandwidth measurement unit 221 measures the bandwidth. In step S282, the control information generation unit 222 generates control information based on the measurement result. In step S283, the control information supply unit 223 supplies the control information to the terminal device 112.

[0190] When the process of step S283 is completed, the transmission control process ends.

[0191] <Flow of Pose Estimation Processing> Furthermore, when performing pose estimation, the server 113 may omit global optimization (BA) using bundle adjustment for some images or a part of it, as in the case of the terminal device 112. An example of the flow of the pose estimation processing in such a case will be described with reference to the flowchart in FIG.

[0192] When the posture estimation process starts, in step S291, the BA execution determination unit 231 determines whether or not to perform bundle adjustment on the captured image to be processed. If it is determined that bundle adjustment is to be performed, the process proceeds to step S292. In step S292, the posture estimation unit 232 performs posture estimation using bundle adjustment. When the process of step S292 ends, the process proceeds to step S294.

[0193] If it is determined in step S291 that bundle adjustment is not to be performed, the process proceeds to step S293. In step S293, the posture estimation unit 233 performs posture estimation without using bundle adjustment. When the process of step S293 ends, the process proceeds to step S294.

[0194] In step S294, the point cloud evaluation unit 234 evaluates the point cloud using the derived camera attitude information. In step S295, the evaluation information generation unit 235 generates evaluation information indicating the evaluation result.

[0195] When the process of step S295 ends, the posture estimation process ends.

[0196] By performing the processes as described above, the server 113 can apply the present technology described above in <2. Transmission Control>. Therefore, posture estimation can be performed more quickly.

[0197] <4. Other Configuration Examples> <High-Precision 3D Compositing Using Camera Pose Information> In the above, we have described 3D composition using selected camera posture information and using the results of that 3D composition for image display and control. However, for example, camera posture information derived using this technology may be applied to other 3D compositions. For example, incremental 3D composition may be performed during imaging, and then batch 3D composition may be performed after imaging is completed. In incremental 3D composition, parameters and processing may be omitted or simplified, or the captured images may be reduced in size to perform 3D composition more quickly. In batch 3D composition, these omissions and simplifications are not performed, and 3D composition with higher precision is performed without these omissions or simplifications. In this series of processes, this technology may be applied to incremental 3D composition, and the obtained camera posture information may be used as the initial value for posture information for posture estimation in batch 3D composition. This can compensate for failures and stabilize the generation process not only in incremental 3D composition using this technology, but also in batch 3D composition using the camera posture information. This batch 3D synthesis is also called high-precision 3D synthesis processing.

[0198] In other words, although the server 113 has been described as performing not only posture estimation but also reconstruction of 3D data, it may be configured to perform only posture estimation (deriving camera posture information and determining the camera posture information to be applied), omit (skip) reconstruction of 3D data, and output (supply) the camera posture information.

[0199] <Synthesis System> An example of the main configuration of a synthesis system in this case is shown in Fig. 30. The 3D synthesis system 300 shown in Fig. 30 includes an imaging device 311, a terminal device 312, a server 313, an information processing device 314, and a 3D synthesis device 315.

[0200] The imaging device 311 is the same as the imaging device 111 described above and performs the same processing. The terminal device 312 is the same as the terminal device 112 described above and performs the same processing. The information processing device 314 is the same as the information processing device 114 described above and performs the same processing. The server 313 performs processing related to the server 113's posture estimation described above and determination of the camera posture information to be applied. The server 313 also performs the posture estimation using a simplified method. This processing is also referred to as simplified posture estimation processing. The server 313 then supplies the derived camera posture information to the 3D composition device 315.

[0201] The 3D synthesizer 315 executes high-precision 3D synthesis processing. That is, the 3D synthesizer 315 executes high-precision pose estimation processing and high-precision 3D modeling processing. In this high-precision pose estimation processing, the 3D synthesizer 315 applies the camera posture information supplied from the server 313 as an initial value. By doing so, the 3D synthesizer 315 can suppress an increase in the processing time for high-precision 3D synthesis processing.

[0202] Even in such a configuration, the present technology described above can be applied to the terminal device 312 and the server 313 in the same manner as in the above example. In other words, by applying the present technology to the 3D synthesis system 300, posture estimation can be performed more quickly.

[0203] <3D Composition System> Furthermore, the imaging device 111, terminal device 112, and display control device 115 of the 3D composition system 100 may be integrated into one unit, as in the 3D composition system 400 shown in Fig. 31 . As shown in Fig. 34 , the 3D composition system 400 has a terminal device 411, a server 412, and an information processing device 413. The server 412 is the same device as the server 113 and performs the same processing. The information processing device 413 is the same device as the information processing device 114 and performs the same processing.

[0204] The terminal device 411 has an imaging unit 421, a transmission control unit 422, and a display control unit 423. The imaging unit 421 has the same configuration as the imaging device 111 and performs the same processing. The transmission control unit 422 has the same configuration as the terminal device 112 and performs the same processing. The display control unit 423 has the same configuration as the display control device 115 and performs the same processing.

[0205] Therefore, even in such a configuration, the present technology described above can be applied in the same manner as in the above example to the transmission control unit 422 and the server 412. In other words, by applying the present technology to the 3D synthesis system 400 as well, posture estimation can be performed more quickly.

[0206] <3D Composition System> For example, the imaging device 111, terminal device 112, and server 113 of the 3D composition system 100 may be integrated into one unit, as in the 3D composition system 500 shown in Fig. 32. As shown in Fig. 36, the 3D composition system 500 has a terminal device 511 and a display control device 512. The display control device 512 is a device similar to the display control device 115, and performs similar processing.

[0207] The terminal device 511 has an imaging unit 521 and a 3D modeling unit 522. The imaging unit 521 has the same configuration as the imaging device 111 and performs the same processing. The 3D modeling unit 522 has the same configuration as the server 113 and performs the same processing.

[0208] In other words, in this example, since the captured image or feature information is not transmitted, control of this transmission is not necessary. However, control of the processing applied within the server 113 is also effective in this example. In other words, the above-described present technology can be applied to the 3D modeling unit 522 in the same manner as in the above-described example. In other words, by applying the present technology to this 3D synthesis system 500, posture estimation can be performed more quickly.

[0209] <3D Composition System> For example, the server 113 and display control device 115 of the 3D composition system 100 may be integrated into one unit, as in a 3D composition system 600 shown in Fig. 33. As shown in Fig. 33, the 3D composition system 600 has an imaging device 611, a terminal device 612, a server 613, and an information processing device 614. The imaging device 611 is the same device as the imaging device 111 and performs the same processing. The terminal device 612 is the same device as the terminal device 112 and performs the same processing. The information processing device 614 is the same device as the information processing device 114 and performs the same processing.

[0210] The server 613 has a 3D modeling unit 621 and a display control unit 622. The 3D modeling unit 621 has the same configuration as the server 113 and performs the same processing. The display control unit 622 has the same configuration as the display control device 115 and performs the same processing.

[0211] Even in such a configuration, the present technology described above can be applied to the terminal device 612 and the 3D modeling unit 621 of the server 613 in the same manner as in the above example. In other words, by applying the present technology to the 3D synthesis system 300 as well, posture estimation can be performed more quickly.

[0212] <3D Composition Device> For example, like the 3D composition device 700 shown in Fig. 34, the imaging device 111, terminal device 112, server 113, and display control device 115 of the 3D composition system 100 may all be integrated into a single device. As shown in Fig. 34, the 3D composition device 700 has an imaging unit 711, a 3D modeling unit 712, and a display control unit 713. The imaging unit 711 has the same configuration as the imaging device 111 and performs the same processing. The 3D modeling unit 712 has the same configuration as the server 113 and performs the same processing. The display control unit 713 has the same configuration as the display control device 115 and performs the same processing.

[0213] That is, in this example, as in the example of FIG. 32 (3D synthesis system 500), no captured image or feature information is transmitted, and therefore no control related to this transmission is necessary. However, control of the processing applied within the server 113 is also effective in this example. That is, the above-described present technology can be applied to the 3D modeling unit 522 in the same manner as in the above-described example. That is, by applying the present technology to this 3D synthesis device 700, posture estimation can be performed more quickly.

[0214] 5. Supplementary Notes Computer The above-described series of processes can be executed by hardware or software. When the series of processes are executed by software, the programs that make up the software are installed on a computer. Here, the term computer includes computers built into dedicated hardware, and general-purpose personal computers, for example, that can execute various functions by installing various programs.

[0215] FIG. 35 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0216] In a computer 1900 shown in FIG. 35, a CPU (Central Processing Unit) 1901, a ROM (Read Only Memory) 1902, and a RAM (Random Access Memory) 1903 are interconnected via a bus 1904.

[0217] An input / output interface 1910 is also connected to the bus 1904. An input unit 1911, an output unit 1912, a storage unit 1913, a communication unit 1914, and a drive 1915 are connected to the input / output interface 1910.

[0218] The input unit 1911 includes, for example, a keyboard, a mouse, a microphone, a touch panel, and an input terminal. The output unit 1912 includes, for example, a display, a speaker, and an output terminal. The storage unit 1913 includes, for example, a hard disk, a RAM disk, and a non-volatile memory. The communication unit 1914 includes, for example, a network interface. The drive 1915 drives removable media 1921 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0219] In a computer configured as described above, the CPU 1901 loads a program stored in the storage unit 1913, for example, into the RAM 1903 via the input / output interface 1910 and the bus 1904, and executes the program. This performs the series of processes described above. The RAM 1903 may store data and the like necessary for the CPU 1901 to execute various processes, as appropriate.

[0220] The program executed by the computer may be applied by being recorded on, for example, removable media 1921 such as package media. In this case, the program may be read from removable media 1921 attached to drive 1915 and installed in storage unit 1913 via input / output interface 1910.

[0221] This program may also be provided via any wired or wireless transmission medium, such as a local area network, the Internet, digital satellite broadcasting, etc. In this case, the program may be received by the communication unit 1914 and installed in the storage unit 1913 via the input / output interface 1910.

[0222] Alternatively, this program may be installed in advance in the ROM 1902 or the storage unit 1913, or both.

[0223] <Application of the Present Technology> The present technology can be applied to any configuration. For example, the present technology can be applied to various electronic devices.

[0224] Furthermore, for example, the present technology can also be implemented as part of an apparatus, such as a processor (e.g., a video processor) as a system LSI (Large Scale Integration), a module using multiple processors (e.g., a video module), a unit using multiple modules (e.g., a video unit), or a set in which other functions are added to a unit (e.g., a video set).

[0225] Furthermore, for example, the present technology can also be applied to a network system configured with multiple devices. For example, the present technology may be implemented as cloud computing in which multiple devices share and collaborate on processing via a network. For example, the present technology may be implemented in a cloud service that provides image (video)-related services to any terminal, such as a computer, an AV (Audio Visual) device, a portable information processing terminal, or an IoT (Internet of Things) device.

[0226] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.

[0227] <Others> In this specification, the term "associate" means, for example, making it possible to use (link) one piece of data when processing the other piece of data. In other words, pieces of data associated with each other may be combined into one piece of data, or may be individual pieces of data. For example, information associated with certain data may be transmitted over a transmission path different from that of the data. Furthermore, for example, information associated with certain data may be recorded on a recording medium different from that of the data (or on a different recording area of ​​the same recording medium). Note that this "association" may refer not to the entire data, but to part of the data. For example, dynamic 3D data and information corresponding to the dynamic 3D data may be associated with each other in any unit, such as multiple frames, one frame, or a portion of a frame.

[0228] In this specification, terms such as "composite," "multiplex," "add," "integrate," "include," "store," "embed," "insert," and the like refer to combining multiple items into one, such as combining encoded data and metadata into one piece of data, and refer to one method of "associating" as described above.

[0229] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present technology.

[0230] For example, a configuration described as one device (or processing unit) may be divided and configured as multiple devices (or processing units). Conversely, configurations described above as multiple devices (or processing units) may be combined and configured as one device (or processing unit). Of course, configurations other than those described above may be added to the configuration of each device (or each processing unit). Furthermore, as long as the configuration and operation of the entire system are substantially the same, part of the configuration of one device (or processing unit) may be included in the configuration of another device (or other processing unit).

[0231] Furthermore, for example, the above-described program may be executed in any device, as long as the device has the necessary functions (functional blocks, etc.) and is able to obtain the necessary information.

[0232] Also, for example, each step of a single flowchart may be executed by a single device, or may be shared and executed by multiple devices. Furthermore, when a single step includes multiple processes, the multiple processes may be executed by a single device, or may be shared and executed by multiple devices. In other words, multiple processes included in a single step can be executed as multiple step processes. Conversely, processes described as multiple steps can be executed collectively as a single step.

[0233] For example, the steps of a program executed by a computer may be executed in chronological order in the order described herein, or may be executed in parallel or individually at the required timing, such as when a call is made. In other words, as long as no contradiction occurs, the steps may be executed in an order different from the order described above. Furthermore, the steps of this program may be executed in parallel with the processing of another program, or may be executed in combination with the processing of another program.

[0234] Furthermore, for example, multiple technologies related to the present technology can be implemented independently and independently, as long as no contradiction occurs. Of course, any multiple technologies can also be implemented in combination. For example, part or all of the present technology described in any embodiment can be implemented in combination with part or all of the present technology described in another embodiment. Furthermore, part or all of any of the above-described present technologies can be implemented in combination with other technologies not described above.

[0235] The present technology may also be configured as follows. (1) An information processing device comprising: a determination unit that determines whether to supply a captured image or feature amount information generated from the captured image to a device that derives posture information indicating the posture of a camera corresponding to the captured image; and a supply unit that supplies the captured image or the feature amount information to the device in accordance with the determination. (2) The information processing device according to (1), further comprising: a measurement unit that measures a bandwidth of a transmission path, wherein the determination unit determines whether to supply the captured image or the feature amount information based on the measured bandwidth. (3) The information processing device according to (1) or (2), wherein the determination unit determines whether to supply the captured image or the feature amount information based on a use of the posture information. (4) The information processing device according to any of (1) to (3), further comprising: an instruction receiving unit that receives a user instruction instructing whether to supply the captured image or the feature amount information, wherein the determination unit determines whether to supply the captured image or the feature amount information based on the received user instruction. (5) The information processing device according to any one of (1) to (4), further comprising a control information acquisition unit that acquires control information for controlling whether to supply the captured image or the feature amount information, wherein the decision unit decides whether to supply the captured image or the feature amount information based on the acquired control information. (6) The information processing device according to any one of (1) to (5), further comprising a captured image selection unit that selects the captured image based on a quality of the captured image, wherein the supply unit supplies the selected captured image when it is decided to supply the captured image. (7) The information processing device according to any one of (1) to (6), further comprising a priority setting unit that sets a priority order for supplying the captured images, wherein the supply unit supplies the captured images according to the priority order. (8) The information processing device according to (7), further comprising the priority setting unit that sets a higher priority order for the captured image having more newly matching feature points.(9) The information processing device according to any one of (1) to (8), further comprising a feature point extraction unit that extracts feature points of the captured image, and when it is determined that the feature amount information should be supplied, the supply unit supplies the extracted feature points as the feature amount information. (10) The information processing device according to (9), further comprising: a feature point determination unit that determines a feature amount of the captured image to be supplied as the feature amount information, and the feature point extraction unit extracts the feature points corresponding to the determined feature amount. (11) The information processing device according to (9) or (10), further comprising: a region-of-interest setting unit that sets a region of interest, and the feature point extraction unit extracts the feature points of the set region of interest. (12) An information processing method, further comprising: determining whether to supply the captured image or feature amount information generated from the captured image to a device that derives posture information indicating the posture of a camera corresponding to the captured image, and supplying the captured image or the feature amount information to the device in accordance with the determination.

[0236] (13) An information processing device comprising: an acquisition unit that acquires captured images or feature amount information generated from the captured images; and a posture information derivation unit that derives posture information indicating the posture of a camera corresponding to the captured images, using the acquired captured images or the feature amount information. (14) The information processing device according to (13), in which the posture information derivation unit sequentially derives the derivable posture information, using the acquired captured images or the feature amount information. (15) The information processing device according to (13), in which the posture information derivation unit acquires all of the captured images or the feature amount information, and then derives all of the posture information collectively. (16) The information processing device according to any of (13) to (15), in which the posture information derivation unit sets a priority order for posture information derivation and derives the posture information according to the priority order. (17) The information processing device according to any of (13) to (16), in which the posture information derivation unit controls, for each captured image, whether to perform bundle adjustment in posture estimation. (18) The information processing device according to (17), wherein the orientation information derivation unit controls whether to perform the bundle adjustment based on the number of newly matching feature points in the captured image. (19) The information processing device according to any of (13) to (18), further comprising: a measurement unit that measures a bandwidth of a transmission path; a control information generation unit that generates control information for controlling whether to supply the captured image or the feature information based on the measured bandwidth; and a control information supply unit that supplies the generated control information to a device that supplies the captured image or the feature information. (20) An information processing method, comprising: acquiring a captured image or feature information generated from the captured image; and deriving orientation information indicating an orientation of a camera corresponding to the captured image using the acquired captured image or the feature information.

[0237] (21) An information processing device comprising: a determination unit that determines for each captured image whether or not to perform bundle adjustment in estimating the attitude of a camera corresponding to the captured image; a first attitude estimation unit that performs the attitude estimation using the bundle adjustment for the captured image for which it is determined that the bundle adjustment should be performed, and derives attitude information and point cloud data of the camera; a second attitude estimation unit that performs the attitude estimation without using the bundle adjustment for the captured image for which it is determined that the bundle adjustment should not be performed, and derives attitude information of the camera; an evaluation unit that evaluates the point cloud data using the attitude information; and a display control unit that controls display of a result of the evaluation. (22) An information processing method comprising: determining for each captured image whether or not to perform bundle adjustment in estimating the attitude of a camera corresponding to the captured image; performing the attitude estimation using the bundle adjustment for the captured image for which it is determined that the bundle adjustment should be performed, and deriving attitude information and point cloud data of the camera; performing the attitude estimation without using the bundle adjustment for the captured image for which it is determined that the bundle adjustment should not be performed, and deriving attitude information of the camera; evaluating the point cloud data using the attitude information; and controlling display of a result of the evaluation.

[0238] 61 Terminal device, 61A Control unit, 62 Server, 62A Control unit, 63 Information processing device, 63A Control unit, 100 3D synthesis system, 111 Imaging device, 112 Terminal device, 113 Server, 114 Information processing device, 115 Display control device, 131 Control unit, 141 Captured image acquisition unit, 142 Transmission information generation unit, 143 Transmission information supply unit, 144 Bandwidth measurement unit, 145 Posture estimation unit, 146 Output unit, 147 Input unit, 148 Control information acquisition unit, 151 Captured image processing unit, 152 Feature amount information generation unit, 161 Captured image reduction unit, 162 Captured image selection unit, 163 Captured image encoding unit, 164 Priority order setting unit, 171 ROI setting unit, 172 Feature point extraction unit 173 Feature point matching unit, 174 Attitude estimation unit, 175 Feature encoding unit, 181 Feature point matching unit, 182 BA execution determination unit, 183 Attitude estimation unit, 184 Attitude estimation unit, 185 Point cloud evaluation unit, 186 Evaluation result display control unit, 201 Control unit, 211 Acquisition unit, 212 Captured image decoding unit, 213 Feature point extraction unit, 214 Feature point matching unit, 215 Attitude estimation unit, 216 Feature amount information decoding unit, 217 Storage unit, 218 Depth image estimation unit, 219 Fusion processing unit, 220 Supply unit, 221 Bandwidth measurement unit, 222 Control information generation unit, 223 Control information supply unit, 231 BA execution determination unit, 232 Attitude estimation unit, 233 Attitude estimation unit 234 Point cloud evaluation unit, 235 Evaluation information generation unit, 300 3D synthesis system, 311 Imaging device, 312 Terminal device, 313 Server, 314 Information processing device, 315 3D synthesis device, 400 3D synthesis system, 411 Terminal device, 412 Server, 413 Information processing device, 421 Imaging unit, 422 Transmission control unit, 423 Display control unit, 500 3D synthesis system, 511 Terminal device, 512 Display control device, 521 Imaging unit, 522 3D modeling unit, 600 3D synthesis system, 611 Imaging device, 612 Terminal device, 613 Server, 614 Information processing device, 621 3D modeling unit, 622 Display control unit, 700 3D synthesis device, 711 Imaging unit, 712 3D modeling unit, 713 display control unit,1900 Computer,

Claims

1. An information processing device comprising: a decision unit that decides whether to supply a captured image or feature information generated from the captured image to a device that derives orientation information indicating the orientation of a camera corresponding to the captured image; and a supply unit that supplies the captured image or the feature information to the device in accordance with the decision.

2. The information processing device according to claim 1, further comprising a measurement unit that measures a bandwidth of a transmission path, wherein the decision unit decides whether to supply the captured image or the feature amount information based on the measured bandwidth.

3. The information processing device according to claim 1, wherein the determination unit determines whether to supply the captured image or the feature amount information based on a purpose of the orientation information.

4. The information processing device of claim 1, further comprising an instruction receiving unit that receives a user instruction indicating whether to supply the captured image or the feature information, and the decision unit decides whether to supply the captured image or the feature information based on the received user instruction.

5. The information processing device according to claim 1, further comprising a control information acquisition unit that acquires control information for controlling whether to supply the captured image or the feature information, and the decision unit decides whether to supply the captured image or the feature information based on the acquired control information.

6. The information processing device according to claim 1, further comprising a captured image selection unit that selects the captured image based on a quality of the captured image, wherein the supply unit supplies the selected captured image when it is determined that the captured image should be supplied.

7. The information processing device according to claim 1, further comprising a priority setting unit that sets a priority order for supplying the captured images, wherein the supply unit supplies the captured images in accordance with the priority order.

8. The information processing device according to claim 7, wherein the priority setting section sets a higher priority to the captured image having a larger number of newly matching feature points.

9. The information processing device according to claim 1, further comprising a feature point extraction unit that extracts feature points of the captured image, wherein the supply unit supplies the extracted feature points as the feature amount information when it is determined that the feature amount information should be supplied.

10. The information processing device according to claim 9, wherein the determination unit further determines a feature of the captured image to be supplied as the feature information, and the feature point extraction unit extracts the feature points corresponding to the determined feature.

11. An information processing device according to claim 9, further comprising an attention area setting unit that sets an attention area, wherein the feature point extraction unit extracts the feature points of the set attention area.

12. An information processing method comprising: determining whether to supply a captured image or feature information generated from the captured image to a device that derives orientation information indicating the orientation of a camera corresponding to the captured image; and supplying the captured image or the feature information to the device in accordance with the determination.

13. An information processing device comprising: an acquisition unit that acquires a captured image or feature information generated from the captured image; and an orientation information derivation unit that uses the acquired captured image or feature information to derive orientation information indicating the orientation of a camera corresponding to the captured image.

14. The information processing device according to claim 13, wherein the posture information derivation unit sequentially derives the posture information that can be derived by using the acquired captured image or the feature amount information.

15. The information processing device according to claim 13, wherein the posture information derivation unit acquires all of the captured images or the feature amount information, and then derives all of the posture information collectively.

16. The information processing device according to claim 13, wherein the posture information derivation unit sets a priority order for posture information derivation and derives the posture information in accordance with the priority order.

17. The information processing device according to claim 13, wherein the posture information derivation unit controls, for each captured image, whether or not to perform bundle adjustment in posture estimation.

18. The information processing device according to claim 17, wherein the posture information derivation unit controls whether or not to perform the bundle adjustment based on the number of newly matching feature points in the captured image.

19. An information processing device as described in claim 13, further comprising: a measurement unit that measures the bandwidth of a transmission path; a control information generation unit that generates control information that controls whether to supply the captured image or the feature information based on the measured bandwidth; and a control information supply unit that supplies the generated control information to a device that supplies the captured image or the feature information.

20. An information processing method comprising: acquiring a captured image or feature information generated from the captured image; and deriving orientation information indicating the orientation of a camera corresponding to the captured image, using the acquired captured image or feature information.