Information processing method, information processing device, and program

Abstract

[Problem] To enable more comprehensive imaging of a subject. [Solution] This information processing method by a computer includes: setting a second imaging condition for imaging a target object or a target area on the basis of a first imaging result obtained by imaging the target object or the target area according to a first imaging condition; and determining an imaging device that performs imaging satisfying the second imaging condition from among a plurality of imaging devices on the basis of at least one of the remaining battery level or available data capacity of the plurality of imaging devices.

Description

Information processing method, information processing device, and program

[0001] The present disclosure relates to an information processing method, an information processing device, and a program.

[0002] 2. Description of the Related Art In recent years, it has become common to generate a product such as a three-dimensional model of a subject by processing a plurality of captured images of the subject.

[0003] For example, Patent Document 1 listed below discloses a technology for generating a three-dimensional model of a subject using images of the subject captured by multiple cameras placed at different positions and camera parameters indicating the position and orientation of each camera.

[0004] International Publication No. 2019 / 225681

[0005] However, after capturing an image of a subject, when generating a product from multiple captured images, it may be discovered that there are areas of the subject that have not been captured. In such cases, it may be difficult to generate a product through image processing, or the accuracy of the product may be reduced.

[0006] Therefore, there is a demand for capturing images of a subject more comprehensively.

[0007] According to the present disclosure, there is provided an information processing method by a computer, which includes setting second imaging conditions for imaging an object or a target area based on a first imaging result in which the object or the target area is imaged in accordance with a first imaging condition, and determining an imaging device from among a plurality of imaging devices that will perform imaging that satisfies the second imaging condition based on at least one of the remaining battery charge or free data capacity of the plurality of imaging devices.

[0008] Furthermore, according to the present disclosure, there is provided an information processing device including: a condition setting unit that sets second imaging conditions for imaging an object or a target area based on a first imaging result obtained by imaging the object or the target area in accordance with a first imaging condition; and an equipment determination unit that determines an imaging device from among a plurality of imaging devices that will perform imaging that satisfies the second imaging condition based on at least one of the remaining battery charge or free data capacity of the plurality of imaging devices.

[0009] Furthermore, according to the present disclosure, there is provided a program for causing a computer to function as: a re-imaging setting unit that sets second imaging conditions for imaging an object or a target area based on a first imaging result obtained by imaging the object or the target area in accordance with first imaging conditions; and an equipment determination unit that determines an imaging device from among a plurality of imaging devices that will perform imaging that satisfies the second imaging conditions based on at least one of the remaining battery charge or free data capacity of the plurality of imaging devices.

[0010] FIG. 1 is a schematic explanatory diagram illustrating generation of a three-dimensional model using the technology according to the present disclosure. FIG. 2 is an explanatory diagram illustrating a method for deriving the shape of an object from a plurality of images based on parallax information. FIG. 3 is a block diagram illustrating a functional configuration of an information processing device according to an embodiment of the present disclosure. FIG. 4 is a schematic explanatory diagram illustrating an example of a display image generated by an image generation unit. FIG. 5 is a flowchart illustrating a flow of operations performed by the information processing device according to the same embodiment. FIG. 6 is an explanatory diagram illustrating a modified example that realizes the technology according to the present disclosure. FIG. 7 is a block diagram illustrating an example hardware configuration of the information processing device according to the same embodiment.

[0011] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0012] The description will be given in the following order: 1. Overview 2. Configuration example of information processing device 3. Operation example of information processing device 4. Modification 5. Hardware configuration example

[0013] 1. Overview First, an overview of the technology according to the present disclosure, including the technical background, will be described with reference to Figures 1 and 2. Figure 1 is a schematic explanatory diagram illustrating generation of a three-dimensional model using the technology according to the present disclosure.

[0014] As shown in Fig. 1, in photogrammetry technology for generating a three-dimensional model of an object Ob from an image, an image of the object Ob is captured using multiple image capturing devices C1 to C3 (referred to as image capturing device C when not distinguished from one another). Note that although Fig. 1 illustrates three image capturing devices C1 to C3, the number of image capturing devices C that capture the object Ob is not limited to the number shown. The number of image capturing devices C that capture the object Ob may be two or less, or may be four or more.

[0015] The object Ob is, for example, a large structure. The object Ob may be a large outdoor structure such as a bridge, a steel tower, or a building, or may be a building or other structure whose three-dimensional structure, both its exterior and interior, is modeled. The object Ob may also be a collection of multiple structures, or may be a predetermined area including multiple structures.

[0016] It is difficult to capture a comprehensive image of the entire object Ob using a single imaging device whose position is fixed. Therefore, a plurality of imaging devices C are used to capture comprehensive images of the entire object Ob.

[0017] The imaging devices C1 and C2 are cameras mounted on mobile bodies D1 and D2, such as drones, that can move freely through the air. The mobile bodies D1 and D2 can move to various positions, angles, and altitudes relative to the object Ob, allowing the mounted imaging devices C1 and C2 to capture images of the object Ob from various positions and orientations. Furthermore, by inputting the imaging location and flight path for the object Ob in advance, the mobile bodies D1 and D2 can automatically capture images of the object Ob using the imaging devices C1 and C2 while autonomously moving around the object Ob.

[0018] The imaging device C3 is, for example, a camera operated by the user U. The imaging device C3 can capture an image of the object Ob from, for example, a location close to the ground where it is difficult for the moving bodies D1 and D2 to enter, or from a location where there are many obstacles. The imaging device C3 may be a digital camera, a mobile terminal with a camera such as a smartphone, or a pole camera in which a camera is attached to the end of an extendable pole.

[0019] Images of the object Ob captured by the multiple imaging devices C are transmitted to, for example, an information processing device 100 operated by a user U. The information processing device 100 processes the multiple images of the object Ob to generate a three-dimensional model of the object Ob using photogrammetry technology. The information processing device 100 may be, for example, a laptop PC (Personal Computer), a tablet terminal, or a smartphone.

[0020] 2 is an explanatory diagram that schematically illustrates a method for deriving the shape of an object Ob from multiple images based on parallax information. As shown in FIG. 2, it is assumed that images P1 to P3 of the object Ob are captured from different positions and orientations by image capture devices C1 to C3.

[0021] In this case, the information processing device 100 first derives a correspondence relationship between the images P1 to P3 by matching the images P1 to P3. This allows the information processing device 100 to estimate the position and orientation of the image capturing devices C1 to C3 that captured the images P1 to P3, respectively, based on the correspondence relationship between the images P1 to P3. Next, the information processing device 100 identifies feature points F1 to F3 on the images P1 to P3 that indicate the same feature point F0 of the object Ob. This allows the information processing device 100 to derive the distances from the positions of the image capturing devices C1 to C3 to the feature point F0 of the object Ob based on the positions and orientations of the image capturing devices C1 to C3 and the parallax of the feature points F1 to F3 on the images P1 to P3. This allows the information processing device 100 to derive information about the shape of the object Ob by deriving the distance to the surface of the object Ob from the distance to each extracted feature point.

[0022] The above-described calculation algorithm performed by the information processing device 100 is also called photogrammetry. Among the above-described calculation algorithms, the algorithm for estimating the positions and orientations of the image capture devices C1 to C3 is also called SfM (Structure from Motion).

[0023] As can be seen from the above, in image processing using photogrammetry, it is important to capture images of the object Ob from all angles, with overlapping images of a size that allows them to be matched with each other.

[0024] However, when capturing images of an object Ob using multiple imaging devices C, the division of the imaging ranges of each imaging device C becomes complicated, which can result in gaps in the imaging range or direction. Furthermore, when capturing images of an object Ob autonomously using mobile bodies D1 and D2 equipped with imaging devices C1 and C2, there is a possibility that an image of the desired quality will not be captured due to disturbances to the mobile bodies D1 and D2 or incorrect settings for exposure, etc. If an image of the desired quality, position, and orientation suitable for SfM processing cannot be obtained, the accuracy of the three-dimensional model of the object Ob generated by SfM processing will be reduced.

[0025] The technology according to the present disclosure has been conceived in view of the above circumstances. The technology according to the present disclosure sets second imaging conditions including information about the position and orientation for imaging the object Ob based on a first imaging result obtained by imaging the object Ob according to a first imaging condition. Furthermore, the technology according to the present disclosure can determine an imaging device from among a plurality of imaging devices that will perform imaging that satisfies the second imaging condition based on at least one of the remaining battery charge or the free data capacity of the plurality of imaging devices.

[0026] According to the technology of the present disclosure, it is possible to prevent missing images from occurring in imaging of the object Ob from all around the object Ob when imaging the object Ob for SfM processing, etc. Furthermore, according to the technology of the present disclosure, it is possible to automatically determine an imaging device that will image an area where missing images occur under a first imaging condition, according to a second imaging condition.

[0027] 2. Configuration Example of Information Processing Device Next, a configuration example of an information processing device according to an embodiment that realizes the technology according to the present disclosure outlined above will be described with reference to Fig. 3. Fig. 3 is a block diagram showing the functional configuration of an information processing device 100 according to an embodiment of the present disclosure.

[0028] As shown in FIG. 3, the information processing device 100 according to this embodiment includes a communication unit 110, an image storage unit 120, an SfM processing unit 130, a condition setting unit 140, a device determination unit 150, a device DB storage unit 160, and an image generation unit 170.

[0029] The communication unit 110 is, for example, a communication interface for transmitting and receiving data between the information processing device 100 and the imaging device C. The communication unit 110 may transmit and receive data between the information processing device 100 and the imaging device C via a communication network such as the Internet. The communication unit 110 may be, for example, a communication card for a wired or wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), or WUSB (Wireless USB). The communication unit 110 may also be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various types of communication. Furthermore, the communication unit 110 may be an external input / output interface for transmitting and receiving data by directly connecting the information processing device 100 and the imaging device C wirelessly or via a wired connection.

[0030] The imaging device C that transmits and receives data to and from the information processing device 100 is at least one or more devices having a function of capturing an image of an object Ob. The imaging device C may be, for example, a camera mounted on an autonomously moving vehicle such as a drone, a digital camera operated by a user U, a mobile device with a camera such as a smartphone, or a pole camera.

[0031] For example, an image captured by imaging device C is transmitted from imaging device C to the information processing device 100. However, in order to increase the transmission speed, a compressed image of the captured image may be transmitted from imaging device C to the information processing device 100. Furthermore, in addition to the captured image, information regarding the position (two-dimensional position or three-dimensional position) and orientation at which the captured image was captured, as well as information regarding the settings of imaging device C at the time of capturing the image, may be transmitted from imaging device C to the information processing device 100. Information indicating the progress of imaging, such as the number of images captured by imaging device C or the number of remaining images to be captured, may be transmitted from imaging device C to the information processing device 100. Information indicating the possibility of continuing imaging, such as the remaining battery charge or available data capacity of imaging device C, may be transmitted from imaging device C to the information processing device 100.

[0032] Furthermore, the imaging device C and the information processing device 100 may transmit and receive information indicating the current position of the imaging device C, the positional relationship between the object Ob and the imaging device C, or the position where the captured image was captured, and share this information with each other.

[0033] The information processing device 100 first receives a first imaging result obtained by imaging the object Ob according to first imaging conditions from the imaging device C via the communication unit 110. The first imaging conditions are conditions that instruct the imaging device C to image the object Ob from all around it at various positions and orientations. The information processing device 100 receives the first imaging result via the communication unit 110, and also receives the first imaging conditions under which the first imaging result was captured.

[0034] For example, if the imaging device C is a camera mounted on a moving body, the first imaging condition includes conditions specifying the position, attitude, and orientation at which the camera will capture an image along a predetermined path of the moving body. The moving body equipped with the camera can autonomously fly around the object Ob along the predetermined path to capture an image of the object Ob at the position, attitude, and orientation specified by the first imaging condition. The position of the camera may be detected, for example, by a Global Navigation Satellite System (GNSS) sensor provided on the moving body. The attitude and orientation of the camera may be detected by a gyro sensor, an acceleration sensor, or an Inertial Measurement Unit (IMU) provided on the moving body. Furthermore, if the moving body and the camera are connected by a gimbal, the relative three-dimensional angles (pitch angle, yaw angle, and roll angle) between the moving body and the camera may be detected by an encoder on the gimbal.

[0035] Furthermore, if the imaging device C is a mobile terminal such as a smartphone, the first imaging condition includes a condition specifying the position and orientation of the mobile terminal when capturing an image of the object Ob. By carrying the mobile terminal and moving it to the position specified by the first imaging condition, the user U can capture an image of the object Ob at the position and orientation specified by the first imaging condition. The position of the mobile terminal may be detected, for example, by a Global Navigation Satellite System (GNSS) sensor provided in the mobile terminal. The orientation of the mobile terminal may be detected, for example, by a gyro sensor, an acceleration sensor, or an inertial measurement unit (IMU) provided in the mobile terminal.

[0036] Furthermore, if the imaging device C is a digital camera or a pole camera, the first imaging condition includes a condition specifying the approximate position and orientation of the imaging device C when capturing an image of the object Ob. The user U can capture an image of the object Ob at the position and orientation specified by the first imaging condition by carrying the imaging device C and moving to the position specified by the first imaging condition. In such a case, a more accurate position and orientation of the imaging device C when capturing an image of the object Ob may be estimated, for example, by a visual positioning system (VPS) using the captured image.

[0037] The first imaging condition may further include various setting conditions such as shutter speed, exposure, aperture (F-number), ISO sensitivity, white balance, and focal length when capturing an image of the object Ob with the imaging device C. If the imaging device C is a camera mounted on a moving object, these various setting conditions may be set based on various sensors such as a distance sensor or an illuminance sensor provided on the moving object or the camera. On the other hand, if the imaging device C is a digital camera, pole camera, or smartphone operated by a user U, these various setting conditions may be set based on input from the user U operating the imaging device C.

[0038] The image storage unit 120 stores captured images received from the imaging device C via the communication unit 110. Specifically, the image storage unit 120 stores a group of images of the object Ob captured under first imaging conditions (first imaging results). As will be described later, the image storage unit 120 further stores a group of images of the object Ob captured under second imaging conditions set based on the first imaging results (second imaging results). The image storage unit 120 may be a data storage device configured, for example, with a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device.

[0039] The SfM processing unit 130 first matches each of the images captured by the imaging device C to each other, thereby estimating the position and orientation at which the image was captured. Specifically, the SfM processing unit 130 matches each of the images of the object Ob included in the first imaging result to each other, thereby estimating the position and orientation at which the image of the object Ob included in the first imaging result was captured. Note that if information regarding the position and orientation at which the image was captured has been obtained from the imaging device C as the first imaging condition, the SfM processing unit 130 can more quickly estimate the position and orientation at which the image of the object Ob was captured by referring to the received information.

[0040] Next, the SfM processing unit 130 estimates the normal vector of the image based on the estimated position and orientation, thereby identifying the surface of the object Ob indicated by the image of the object Ob included in the first imaging result. This allows the SfM processing unit 130 to determine the range of the object Ob covered by the image included in the first imaging result. Note that "the range of the object Ob is covered by the image" means that the range of the object Ob is included in the angle of view of the captured image.

[0041] The condition setting unit 140 sets second imaging conditions that instruct the additional capturing of an image of the object Ob based on the range of the object Ob covered in the image included in the first imaging result. Specifically, the condition setting unit 140 may set second imaging conditions that instruct the capturing of an image of a range of the object Ob that is determined not to be covered in the image included in the first imaging result. That is, the second imaging conditions set by the condition setting unit 140 include conditions that instruct the position and orientation of the imaging device C for capturing an image of the range of the object Ob that is determined not to be covered in the image included in the first imaging result.

[0042] For example, when generating a three-dimensional model of the object Ob by photogrammetry using the first imaging result, it is difficult to generate a three-dimensional model for areas not covered by the images included in the first imaging result. Furthermore, areas where the images included in the first imaging result have little overlap may result in low accuracy of the generated three-dimensional model. Therefore, the condition setting unit 140 can instruct the imaging device C to additionally capture areas that are insufficient in the first imaging result when generating a three-dimensional model using the second imaging conditions.

[0043] Therefore, the condition setting unit 140 may set a second imaging condition that instructs capturing an image that covers a range of the object Ob that was determined not to be captured in the first imaging result. Also, the condition setting unit 140 may set a second imaging condition that instructs capturing an image that covers a range of the object Ob in which the overlap of the images included in the first imaging result is smaller than a threshold.

[0044] Furthermore, the image included in the first imaging result may be out of focus, overexposed, or underexposed depending on the setting conditions such as shutter speed, exposure, aperture (F-number), or focal length. It is difficult to extract feature points from an out-of-focus, overexposed, or underexposed image, making it difficult to use the image for image processing such as SfM. In such a case, the condition setting unit 140 may set second imaging conditions that are changed from the first imaging conditions so that the above-mentioned out-of-focus, overexposed, or underexposed condition does not occur. This allows the imaging device C to capture a clearer image of the object Ob under the second imaging conditions.

[0045] The device determination unit 150 determines an imaging device C that will capture images that satisfy the second imaging condition from among the multiple imaging devices C. Specifically, the device determination unit 150 determines an imaging device C that will capture images that satisfy the second imaging condition from among the multiple imaging devices C, based on at least one of the remaining battery power or the free data capacity of the multiple imaging devices C. Note that, if the imaging device C is a camera mounted on a mobile object, the remaining battery power refers to the remaining battery power of both the mobile object and the camera.

[0046] For example, the device determination unit 150 may determine the imaging device C that will perform imaging that satisfies the second imaging conditions based on whether or not there is a remaining battery charge sufficient to perform imaging at the position and orientation specified by the second imaging conditions. If the imaging device C is a camera mounted on a moving object, the remaining battery charge sufficient to perform imaging is a remaining battery charge that is greater than the sum of the amount of power consumed by the moving object when traveling back and forth between its current position and the position specified by the second imaging conditions and the amount of power consumed when imaging with the camera.

[0047] Furthermore, the device determination unit 150 may determine the imaging device C that performs imaging that satisfies the second imaging conditions based on whether or not there is remaining free data capacity that can store images acquired by imaging specified by the second imaging conditions. The free data capacity that can store images is free data capacity that is greater than the data amount obtained by multiplying the number of images acquired by imaging specified by the second imaging conditions by the data amount per image.

[0048] Furthermore, the device determination unit 150 may determine an imaging device C that will perform imaging that satisfies the second imaging condition, based on whether the imaging device C can reach the position and orientation indicated by the second imaging condition. For example, if the imaging device C is a camera mounted on a mobile object and a route to the position and orientation indicated by the second imaging condition is plannable, the device determination unit 150 may determine that the imaging device C can reach the position and orientation indicated by the second imaging condition. However, if the imaging device C is a camera mounted on an air vehicle such as a drone, being able to plan the route means that the planned route is within a flyable area controlled by, for example, unmanned aircraft system traffic management (UTM) or a geofence, and that the route is passable by the size of the mobile object.

[0049] The device determination unit 150 can determine the imaging device C that will capture images that satisfy the second imaging condition based on at least one of the remaining battery charge and the available data capacity, and by comprehensively determining the other factors described above. Note that the device determination unit 150 can also determine multiple imaging devices C as the imaging device C that will capture images that satisfy the second imaging condition.

[0050] Information regarding the position and orientation indicated by the second imaging condition is transmitted to the imaging device C, which has been determined by the device determination unit 150 to perform imaging that satisfies the second imaging condition, via the communication unit 110. Furthermore, if the imaging device C is a camera mounted on a moving object, information regarding a route to reach the position and orientation indicated by the second imaging condition may be further transmitted to the imaging device C.

[0051] The device DB storage unit 160 stores various information about the imaging device C capable of capturing an image of the object Ob. For example, the device DB storage unit 160 may store the model name and camera specifications of the imaging device C capable of capturing an image of the object Ob. If the imaging device C is a camera mounted on a moving object, the device DB storage unit 160 may further store specifications of the moving object, such as the moving speed, maximum reachable altitude, or attitude stability.

[0052] The device DB storage unit 160 may further store information about the current status of the imaging devices C capable of capturing images of the object Ob, such as the current remaining battery charge, free data capacity, or current location. This information is acquired in real time from each of the imaging devices C via the communication unit 110 and stored in the device DB storage unit 160.

[0053] The image generation unit 170 generates a display image showing the correspondence between the object Ob and the first imaging result. The display image generated by the image generation unit 170 is presented to the user U by being output to, for example, a display device 200 connected to the information processing device 100. Specifically, the image generation unit 170 may generate a display image in which the range covered by the image included in the first imaging result is superimposed as an AR (Augmented Reality) image on a through image of the object Ob. This allows the user U to visually confirm the range of the object Ob that is not covered by the image included in the first imaging result by viewing the display image.

[0054] Furthermore, the image generating unit 170 may further superimpose an image showing information about at least one of the remaining battery charge and the free data capacity of each of the imaging devices C that capture the object Ob on the display image showing the correspondence between the object Ob and the first imaging result. In this way, by viewing the display image, the user U can visually confirm which imaging devices C are capable of capturing images that satisfy the second imaging conditions.

[0055] Furthermore, the image generating unit 170 may further superimpose an image showing a range captured under the second imaging conditions on the display image showing the correspondence between the object Ob and the first imaging result. By viewing the display image, the user U can visually confirm that the range not covered in the image included in the first imaging result is being covered in the imaging result captured under the second imaging conditions.

[0056] 4 shows an example of a display image generated by the image generating unit 170. FIG. 4 is a schematic explanatory diagram showing an example of a display image generated by the image generating unit 170.

[0057] As shown in FIG. 4, the display image Pc includes an AR area Pc1 in which a through image Po of the object Ob is displayed, and an information area Pc2 in which information related to the imaging device C is displayed.

[0058] In the AR area Pc1, for example, a range A1 covered by the image included in the first imaging result is displayed superimposed on a through image Po of the object Ob. In Fig. 4, the range A1 is superimposed on one side of the through image Po of the object Ob. In addition, in the AR area Pc1, moving bodies D1 and D2 equipped with image capturing devices C1 and C2, and an image capturing device C3 are displayed at their current positions as actual images or icons.

[0059] Furthermore, when the moving body D1 equipped with the imaging device C1 is instructed to take an image that satisfies the second imaging condition, the flight path Rt of the moving body D1 may be displayed superimposed on the through image Po. By flying along the flight path Rt and taking an image that satisfies the second imaging condition, the moving body D1 can capture an image of the range of the object Ob that is not covered by the range A1 with the imaging device C1.

[0060] In the information area Pc2, for example, an image is displayed showing the remaining operating time derived from the remaining battery charge of each of the imaging devices C capable of capturing images of the object Ob and the number of images that can be captured derived from the free data capacity. The information in the information area Pc2 may be updated in real time based on, for example, information received from the imaging devices C.

[0061] The information processing device 100 having the above configuration can capture additional images of areas where there is a gap in an image of the object Ob captured from all around. The information processing device 100 can also determine the imaging device C that will capture the area where there is a gap based on at least one of the remaining battery charge and the available data capacity.

[0062] In the above description, the first and second imaging results of the object Ob stored in the image storage unit 120 are used for photogrammetry to generate a three-dimensional model of the object Ob. However, the technology disclosed herein is not limited to the above example. The first and second imaging results of the object Ob may be used to generate a deliverable by image processing other than photogrammetry. For example, the first and second imaging results of the object Ob may be used to generate a two-dimensional image (such as a panoramic image or a bird's-eye view image) of the object Ob by stitching images together. Furthermore, the first and second imaging results of the object Ob may be used to generate a three-dimensional model of the object Ob using Neural Radiance Fields (NeRF) technology.

[0063] 3. Example of Operation of Information Processing Apparatus> Next, an example of operation of the information processing apparatus according to this embodiment will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the flow of operations executed by the information processing apparatus 100 according to this embodiment.

[0064] 5, first, an image of an object Ob is captured in accordance with a first imaging condition by each of the imaging devices C (S101). The first imaging condition is, for example, an imaging condition set by a user U in order to capture an image of the entire periphery of the object Ob.

[0065] Next, the information processing device 100 receives first imaging results captured under the first imaging conditions from each of the imaging devices C, and performs SfM processing based on the first imaging results (S102). As a result, the information processing device 100 identifies the range of the object Ob covered by the image included in the first imaging results.

[0066] Next, the information processing device 100 determines whether there is a range of the object Ob to be additionally imaged based on the range of the object Ob covered by the image included in the first imaging result (S103). For example, if there is a range not covered by the image included in the first imaging result, or if there is a range where the images included in the first imaging result have little overlap, the information processing device 100 determines that there is a range of the object Ob to be additionally imaged (S103 / YES). On the other hand, if the range of the object Ob is entirely covered by the image included in the first imaging result, the information processing device 100 determines that there is no range of the object Ob to be additionally imaged (S103 / NO) and terminates operation.

[0067] If it is determined that there is an additional range of the object Ob to be imaged (YES in S103), the information processing device 100 sets a second imaging condition that instructs the image processing device 100 to capture an image that covers an area not covered by the image included in the first imaging result (S104). For example, the information processing device 100 may set the second imaging condition that instructs the imaging position and orientation of the image that covers an area not covered by the image included in the first imaging result.

[0068] Thereafter, the information processing device 100 determines whether or not there is a moving object that can reach the position and orientation specified by the second imaging condition among the moving objects equipped with the imaging device C (S105). Next, the information processing device 100 determines whether or not the remaining battery power of the moving object that can reach the position and orientation specified by the second imaging condition (S105 / YES) is sufficient to perform imaging (S106). Next, the information processing device 100 determines whether or not the free data capacity of the moving object with sufficient remaining battery power (S106 / YES) is sufficient to store the image acquired by imaging (S107).

[0069] If a moving object exists that satisfies all of the determinations in steps S105 to S107 (YES in S105, YES in S106, YES in S107), the information processing device 100 determines the imaging device C mounted on the moving object as the imaging device C that will perform imaging that satisfies the second imaging condition (S108). The imaging device C that has been determined to perform imaging that satisfies the second imaging condition can perform imaging in accordance with the second imaging condition by autonomously flying the moving object (S110).

[0070] On the other hand, if there is no moving object that satisfies all of the determinations in steps S105 to S107 (NO in at least one of S105, S106, and S107), the information processing device 100 determines the imaging device C that can be operated by the user U as the imaging device C that will capture images that satisfy the second imaging conditions (S109). This allows the user U to manually move and operate the imaging device C to capture images in accordance with the second imaging conditions (S110).

[0071] After the image capture under the second imaging conditions has been performed, the information processing device 100 returns to the operation of step S102 and performs SfM processing based on the second imaging results captured under the second imaging conditions in addition to the first imaging results (S102). This allows the information processing device 100 to identify the range of the object Ob covered by the images included in the first imaging results and the second imaging results. By repeating the operations of steps S102 to S110, the information processing device 100 can acquire an image capturing the entire periphery of the object Ob without omitting any part of it.

[0072] 4. Modifications Next, a modification that realizes the technology according to the present disclosure will be described with reference to Fig. 6. Fig. 6 is an explanatory diagram that explains a modification that realizes the technology according to the present disclosure.

[0073] As shown in Figure 6, in a modified example that realizes the technology disclosed herein, there is no information processing device 100 that receives images captured by imaging devices C1, C2, C3, C4, and C5 and centrally performs SfM processing, etc.

[0074] 3 may be provided in mobile objects D1 and D2 equipped with imaging devices C1 and C2, or in an imaging device C5 such as a smartphone. These mobile objects D1 and D2 or imaging device C5 have relatively high processing capabilities and are therefore capable of executing the functions of the information processing device 100. On the other hand, imaging device C3 such as a digital camera and imaging device C4 such as a pole camera are specialized for the single function of imaging, and therefore may not have the functions of the information processing device 100.

[0075] The mobile bodies D1 and D2 equipped with the imaging devices C1 and C2, or the imaging device C5, may execute the functions of the information processing device 100 in parallel with each other, or may execute the functions of the information processing device 100 in a distributed manner.

[0076] When the functions of the information processing device 100 are executed in parallel, redundancy is ensured, so that even if communication with either the mobile bodies D1 and D2 equipped with the image capturing devices C1 and C2 or the image capturing device C5 is interrupted, the information processing according to the present disclosure can be executed continuously. On the other hand, when the functions of the information processing device 100 are executed in a distributed manner, the load of the calculation processing can be distributed, so that the information processing according to the present disclosure can be executed at a higher speed.

[0077] 5. Example of Hardware Configuration The hardware configuration of the information processing device 100 according to this embodiment will be described further with reference to Fig. 7. Fig. 7 is a block diagram showing an example of the hardware configuration of the information processing device 100 according to this embodiment.

[0078] The functions of the information processing device 100 according to this embodiment may be realized by cooperation between software and hardware described below. The functions of the SfM processing unit 130, the condition setting unit 140, the device determination unit 150, and the image generation unit 170 may be executed by, for example, the CPU 901. The functions of the communication unit 110 may be executed by, for example, the connection port 910 or the communication device 911. The functions of the image storage unit 120 and the device DB storage unit 160 may be executed by, for example, the storage device 908.

[0079] As shown in FIG. 7, the information processing device 100 includes a CPU (Central Processing Unit) 901 , a ROM (Read Only Memory) 902 , and a RAM (Random Access Memory) 903 .

[0080] The information processing device 100 may further include a host bus 904a, a bridge 904, an external bus 904b, an interface 905, an input device 906, an output device 907, a storage device 908, a drive 909, a connection port 910, or a communication device 911. The information processing device 100 may have a processing circuit such as a DSP (Digital Signal Processor) or an ASIC (Application Specific Integrated Circuit) instead of or together with the CPU 901.

[0081] The CPU 901 functions as an arithmetic processing device or a control device, and controls operations within the information processing device 100 in accordance with various programs recorded in the ROM 902, the RAM 903, the storage device 908, or a removable recording medium attached to the drive 909. The ROM 902 stores programs used by the CPU 901, calculation parameters, etc. The RAM 903 temporarily stores programs used in the execution of the CPU 901, and parameters used during the execution of the programs.

[0082] The CPU 901, ROM 902, and RAM 903 are interconnected by a host bus 904a capable of high-speed data transmission. The host bus 904a is connected to an external bus 904b, such as a PCI (Peripheral Component Interconnect / Interface) bus, via a bridge 904. The external bus 904b is connected to various components via an interface 905.

[0083] The input device 906 is a device that accepts input from a user, such as a mouse, keyboard, touch panel, button, switch, or lever. The input device 906 may also be a microphone that detects the user's voice. The input device 906 may also be, for example, a remote control device that uses infrared rays or other radio waves, or may be an externally connected device that supports operation of the information processing device 100.

[0084] The input device 906 further includes an input control circuit that outputs an input signal generated based on information input by the user to the CPU 901. By operating the input device 906, the user can input various data to the information processing device 100 or instruct the information processing device 100 to perform processing operations.

[0085] The output device 907 is a device that can visually or audibly present information acquired or generated by the information processing device 100 to a user. The output device 907 may be, for example, a display device such as an LCD (Liquid Crystal Display), a PDP (Plasma Display Panel), an OLED (Organic Light Emitting Diode) display, a hologram, or a projector, a sound output device such as a speaker or headphones, or a printing device such as a printer. The output device 907 may output information acquired by processing of the information processing device 100 as video such as text or an image, or as sound such as voice or audio.

[0086] The storage device 908 is a data storage device configured as an example of a storage unit of the information processing device 100. The storage device 908 may be configured, for example, by a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage device 908 can store programs executed by the CPU 901, various data, various data acquired from the outside, and the like.

[0087] The drive 909 is a device for reading or writing data from or to a removable recording medium such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and is built into or externally attached to the information processing device 100. For example, the drive 909 can read information recorded on an attached removable recording medium and output the information to the RAM 903. The drive 909 can also write data to an attached removable recording medium.

[0088] The connection port 910 is a port for directly connecting an external device to the information processing device 100. The connection port 910 may be, for example, a Universal Serial Bus (USB) port, an IEEE 1394 port, or a Small Computer System Interface (SCSI) port. The connection port 910 may also be an RS-232C port, an optical audio terminal, or a High-Definition Multimedia Interface (HDMI) (registered trademark) port. By connecting the connection port 910 to an external device, various types of data can be transmitted and received between the information processing device 100 and the external device.

[0089] The communication device 911 is, for example, a communication interface configured with a communication device for connecting to the communication network 920. The communication device 911 may be, for example, a communication card for a wired or wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), or WUSB (Wireless USB). The communication device 911 may also be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various types of communication.

[0090] The communication device 911 can transmit and receive signals, for example, via the Internet or other communication devices using a predetermined protocol such as TCP / IP. The communication network 920 connected to the communication device 911 is a wired or wireless network, and may be, for example, an Internet communication network, a home LAN, an infrared communication network, a radio wave communication network, or a satellite communication network.

[0091] It is also possible to create a program that causes hardware such as the CPU 901, ROM 902, and RAM 903 built into a computer to perform functions equivalent to those of the information processing device 100. It is also possible to provide a computer-readable recording medium on which the program is recorded.

[0092] Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical idea described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0093] For example, in the above embodiment, the object Ob imaged under the first and second imaging conditions is a large structure, but the present technology is not limited to this example. For example, the object Ob imaged under the first and second imaging conditions may be a predetermined area such as land or a location.

[0094] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0095] Note that the following configurations also fall within the technical scope of the present disclosure. (1) An information processing method by a computer, comprising: setting second imaging conditions for imaging an object or a target area based on first imaging results obtained by imaging the object or a target area according to first imaging conditions; and determining an imaging device from among a plurality of imaging devices that will perform imaging that satisfies the second imaging conditions based on at least one of the remaining battery power or free data capacity of the plurality of imaging devices. (2) The information processing method described in (1), in which the first imaging conditions and the second imaging conditions include information about the object or target area for which imaging is to be performed. (3) The information processing method described in (2), in which the second imaging conditions include information about the range of the object or target area for which additional imaging is determined based on the first imaging results. (4) The information processing method according to (3), wherein the captured image included in the first imaging result is used to generate a product by image processing, and the range of the object or target area to be additionally imaged under the second imaging conditions is determined based on whether the product can be generated or the quality of the product. (5) The information processing method according to any one of (1) to (4), wherein the first imaging conditions include information on imaging settings. (6) The information processing method according to any one of (1) to (5), wherein the first imaging conditions are determined based on input from a user. (7) The information processing method according to (6), wherein imaging according to the first imaging conditions is performed at least by an imaging device mounted on a moving object, and the first imaging conditions are determined based on output from a sensor provided in the moving object or the imaging device. (8) The information processing method according to any one of (1) to (7), wherein the second imaging conditions include information on imaging settings. (9) The information processing method according to any one of (1) to (8), wherein the second imaging condition includes information about an imaging position at which the target object or target area is imaged. (10) The information processing method according to (9), wherein the second imaging condition further includes information about an orientation of an imaging device at the imaging position.(11) The information processing method according to (9) or (10), wherein the imaging that satisfies the second imaging condition is performed at least by an imaging device mounted on a moving body, and the imaging device that performs the imaging that satisfies the second imaging condition is determined further based on the reachability of the moving body to the imaging position. (12) The information processing method according to (11), wherein the reachability of the moving body to the imaging position is determined based on the remaining battery power, body size, or movable area of ​​the moving body. (13) The information processing method according to (11) or (12), wherein the second imaging condition further includes information on a movement path of the moving body to reach the imaging position. (14) The information processing method according to any one of (1) to (13), further including outputting a display image indicating a correspondence between the object and the first imaging result to a display device. (15) The information processing method according to (14), wherein an image indicating the range of the object or object area imaged under the second imaging condition is further superimposed on the display image. (16) The information processing method according to (14) or (15), wherein an image showing information on at least one of the remaining battery power or the free data capacity of the plurality of imaging devices is further superimposed on the display image. (17) The information processing method according to any one of (14) to (16), wherein an image showing current positions of the plurality of imaging devices or a moving body equipped with the plurality of imaging devices is further superimposed on the display image. (18) An information processing device comprising: a condition setting unit that sets second imaging conditions for imaging an object or a target area based on a first imaging result of imaging the object or the target area in accordance with first imaging conditions; and a device determination unit that determines an imaging device from the plurality of imaging devices that will perform imaging that satisfies the second imaging conditions, based on at least one of the remaining battery power or the free data capacity of the plurality of imaging devices.(19) A program for causing a computer to function as: a re-imaging setting unit that sets second imaging conditions for imaging an object or a target area based on a first imaging result of imaging the object or the target area according to first imaging conditions; and an equipment determination unit that determines an imaging device that will perform imaging that satisfies the second imaging conditions from among a plurality of imaging devices based on at least one of the remaining battery charge or free data capacity of the plurality of imaging devices.

[0096] REFERENCE SIGNS LIST 100 Information processing device 110 Communication unit 120 Image storage unit 130 SfM processing unit 140 Condition setting unit 150 Device determination unit 160 Device DB storage unit 170 Image generation unit 200 Display device U User Ob Object D1, D2 Moving object C1, C2, C3, C4, C5 Imaging device

Claims

1. A computer-implemented information processing method comprising: setting second imaging conditions for imaging an object or target area based on a first imaging result in which the object or target area is imaged in accordance with first imaging conditions; and determining, from among a plurality of imaging devices, an imaging device that will perform imaging that satisfies the second imaging conditions based on at least one of the remaining battery charge or free data capacity of the plurality of imaging devices.

2. The information processing method according to claim 1, wherein the first imaging condition and the second imaging condition include information about the object or target area for which imaging is performed.

3. An information processing method as described in claim 2, wherein the second imaging conditions include information regarding the range of the object or target area for which additional imaging has been determined based on the first imaging results.

4. An information processing method as described in claim 3, wherein the captured image included in the first imaging result is used to generate a product by image processing, and the range of the object or target area for which additional imaging is performed under the second imaging conditions is determined based on whether or not the product can be generated or its quality.

5. The information processing method according to claim 1, wherein the first imaging condition includes information relating to imaging settings.

6. The information processing method according to claim 1, wherein the first imaging condition is determined based on an input from a user.

7. An information processing method according to claim 6, wherein the imaging in accordance with the first imaging conditions is performed at least by an imaging device mounted on a moving body, and the first imaging conditions are determined based on output from a sensor provided on the moving body or the imaging device.

8. The information processing method according to claim 1, wherein the second imaging condition includes information relating to imaging settings.

9. The information processing method according to claim 1, wherein the second imaging condition includes information about an imaging position for imaging the target object or target area.

10. The information processing method according to claim 9, wherein the second imaging condition further includes information regarding the orientation of the imaging device at the imaging position.

11. An information processing method as described in claim 9, wherein the imaging that satisfies the second imaging condition is performed at least by an imaging device mounted on a moving body, and the imaging device that performs the imaging that satisfies the second imaging condition is determined further based on the reachability of the moving body to the imaging position.

12. The information processing method according to claim 11, wherein the reachability of the mobile object to the imaging position is determined based on the remaining battery charge, the size of the mobile object, or the movable area of ​​the mobile object.

13. The information processing method according to claim 11, wherein the second imaging condition further includes information regarding a travel path of the moving object to reach the imaging position.

14. The information processing method according to claim 1, further comprising outputting to a display device a display image showing the correspondence between the object and the first imaging result.

15. An information processing method according to claim 14, wherein an image showing the range of the object or object area imaged under the second imaging conditions is further superimposed on the displayed image.

16. The information processing method according to claim 14, wherein an image showing information about at least one of the remaining battery charge and the free data capacity of the plurality of imaging devices is further superimposed on the display image.

17. The information processing method according to claim 14, wherein an image showing the current positions of the plurality of imaging devices or a mobile object equipped with the plurality of imaging devices is further superimposed on the display image.

18. An information processing device comprising: a condition setting unit that sets second imaging conditions for imaging an object or target area based on a first imaging result obtained by imaging the object or target area according to a first imaging condition; and a device determination unit that determines an imaging device from among the plurality of imaging devices that will perform imaging that satisfies the second imaging condition based on at least one of the remaining battery charge or free data capacity of the plurality of imaging devices.

19. A program for causing a computer to function as: a re-imaging setting unit that sets second imaging conditions for imaging an object or target area based on a first imaging result obtained by imaging the object or target area in accordance with first imaging conditions; and an equipment determination unit that determines, from among a plurality of imaging devices, an imaging device that will perform imaging that satisfies the second imaging conditions based on at least one of the remaining battery charge or free data capacity of the plurality of imaging devices.

Images