Information processing device, information processing method, and program

Abstract

Achieve the purpose of utilizing mobility. [Solution] A receiving unit 121 receives basic requirements including an imaging target area, which is an area to be imaged by a moving object 2, a schedule for performing the imaging, and service level requirements including a delivery date for a 3D structure reconstructed from the captured images and the quality of the 3D structure. A generating unit 122 generates an acquisition plan on the condition that the quality of the basic requirements is satisfied. A determining unit 123 determines a 3D reconstruction plan for each execution unit of imaging, including the acquisition plan, the imaging start time, the transmission time slot for the acquired images captured by the moving object 2, and at least one of the processing priority determination and computational resource allocation of the server 3, based on (1) each acquisition plan, (2) movement constraints within each imaging target area, and (3) at least one of available network resources and computational resources of the server 3 that performs the reconstruction, on the condition that each delivery date is satisfied, for each execution unit of imaging defined in each schedule.

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, unmanned mobile objects such as drones (hereinafter referred to simply as "mobile objects" in this specification) have become increasingly common, and technologies for utilizing mobile objects have also been developing. For example, Patent Document 1 discloses a technology that dynamically considers and comprehensively determines the noise, vibration, and possibility of falling objects that may occur while a mobile object is moving, and calculates a travel route that minimizes the probability of noise, vibration, and falling objects occurring. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-088422 Summary of the Invention [Problem to be solved by the invention]

[0004] In order to utilize a mobile object more effectively, the mobile object may be used for multiple purposes and multiple travel routes may be set. In such a case, even if a utilization plan for the mobile object is formulated with only a single utilization purpose in mind, there is a risk that the multiple utilization purposes of the mobile object may not be fully achieved.

[0005] Therefore, the present invention has been made in consideration of these points, and aims to provide a technology that can achieve the purpose of utilizing mobile objects. [Means for solving the problem]

[0006] A first aspect of the present invention is an information processing device, which includes a receiving unit that receives basic requirements including an imaging target area that is an area that a moving body will image, a schedule including a period during which the moving body will perform imaging and an interval at which the moving body will repeat imaging within that period, and service level requirements including a delivery time for providing a three-dimensional structure reconstructed from images captured by the moving body and quality related to the positional accuracy of the three-dimensional structure after reconstruction, a generating unit that generates an acquisition plan including a path of the moving body on the condition that the quality is satisfied for the basic requirements, and a generating unit that generates, for each of the imaging execution units defined in the schedule of a plurality of the basic requirements, each of the service level requirements included in the service level requirements corresponding to each basic requirement. The system includes a determination unit that determines a three-dimensional reconstruction plan for each of the execution units, including the acquisition plan, the start time of imaging of the moving body, the transmission time period of the acquired image captured by the moving body, and at least one of determining the processing priority of the server and allocating the computational resources of the server, based on (1) each of the acquisition plans, (2) movement constraints including at least one of the time periods and areas in which the moving body can move, which are set in each of the imaging target areas, and (3) at least one of the network resources of a hangar that stores the moving body and the computational resources of a server that reconstructs the three-dimensional structure, provided that the delivery deadline is met.

[0007] When the determination unit identifies that there is a delivery date that cannot be met among the delivery dates associated with each of the multiple execution units, the generation unit may modify the acquisition plan for the execution unit associated with that delivery date so as to shorten the time required to execute the acquisition plan, provided that the service level requirement is met, and the determination unit may re-determine the three-dimensional reconstruction plan for the execution unit using the modified acquisition plan.

[0008] The determination unit may be further based on the network resources for each communication network, with the further condition that the maximum value of the network resource usage per specified unit time for each communication network is smaller, and may determine the transmission time period for the acquired image captured by the moving body and the communication network to be used for that transmission time period.

[0009] The determination unit may further base its determination on cost information including the usage costs of the network resources for each communication network, with the further condition that the total usage costs of the network resources within a specified period is smaller, and may determine the transmission time period of the acquired image captured by the mobile body and the communication network to be used during that transmission time period.

[0010] The determination unit may determine at least one of the processing priority of the server and the allocation of the server's computing resources, with the further condition that the maximum amount of computing resource usage of the server per specified unit time is smaller.

[0011] The determination unit may use at least one of the maximum amount of usage of the network resources per predetermined unit time, the maximum amount of usage of the server's computational resources per predetermined unit time, the cost of compensation to customers for not satisfying the service level requirements, the total execution costs of the three-dimensional reconstruction plans, the variance of the margin indicated by the difference between the actual delivery amount for each of the imaging target areas and the delivery date, and the variance of the margin indicated by the difference between the actual quality value for each of the imaging target areas and the quality included in the service level requirements as an objective function, and may use the service level requirements, the available time slots, the available area, the upper limit of the network resources for each communication network, the unit price of network resources for each communication network, and the upper limit of the server's computational resources as constraints, and may determine the three-dimensional reconstruction plan by solving an optimization problem so as to minimize the objective function for each of the execution units defined in the schedule of the basic requirements.

[0012] The determination unit may determine the three-dimensional reconstruction plan, which further includes an allocation of the mobile bodies to be used in the acquisition plan, the time periods during which the mobile bodies are charged, and the time periods during which the mobile bodies are to replace their batteries, based on the number of mobile bodies available for imaging, the number of mobile bodies that can be charged simultaneously, the number of available batteries, and the number of mobile bodies that can be stored in the hangar, with the further condition that the distance between two different mobile bodies is greater than a predetermined distance when the mobile bodies depart or arrive at the hangar.

[0013] The information processing device may further include a difference acquisition unit that acquires difference image data from an acquired image captured in accordance with the acquisition plan included in the three-dimensional reconstruction plan and an image reconstructed in relation to the past execution unit, and a transmission unit that transmits the difference image data to the server.

[0014] Before generating the differential image data, the difference acquisition unit may perform at least one of a geometric alignment process that geometrically aligns the images from which the differential image data is to be generated based on a common coordinate system, and an optical alignment process that corrects differences in exposure or white balance.

[0015] The information processing device may further include an encoding unit that encodes the acquired images using an encoding profile from among a plurality of encoding profiles that relatively shortens the transmission time of the acquired images captured in accordance with the acquisition plan included in the three-dimensional reconstruction plan, provided that the quality is satisfied, and a transmission unit that transmits the encoded acquired images to the server.

[0016] The generation unit may generate the acquisition plan further including an imaging density, and the information processing device may further include a prediction unit that predicts, for each of the three-dimensional reconstruction plans, a remaining time obtained by subtracting the total time predicted to be required to execute unfinished processes among the execution of the acquisition plan by the moving body, the transmission processing of the acquired images, and the reconstruction processing by the server from the time until the delivery date. When the prediction unit identifies that there is a three-dimensional reconstruction plan for which the remaining time is shorter than a predetermined threshold, the determination unit may perform at least one of the following, on condition that the service level requirements are satisfied: (1) instruct the generation unit to make a correction to reduce the number of captured images to be acquired in the acquisition plan included in the identified three-dimensional reconstruction plan, or to shorten the time required to execute the acquisition plan; or (2) for a three-dimensional reconstruction plan whose transmission time zone overlaps with the transmission time zone of the identified three-dimensional reconstruction plan, modify the three-dimensional reconstruction plan so that the overlap of the transmission time zones is reduced.

[0017] The generation unit may divide the imaging target area into a plurality of partial areas, and may generate the route of the moving body included in the acquisition plan as a series of routes that are closed-loop routes with each partial area being an imaging target, and that can generate a point cloud for each partial area that is sparser than the three-dimensional structure provided using only images captured from the route, and the information processing device may further include a transmission unit that transmits the acquired images to the server in sequence in units of partial areas, and a reconstruction startup unit that causes the server to independently start reconstruction pre-processing for generating the point cloud for each partial area.

[0018] The information processing device may further include a memory unit that stores a learning model that has been trained to output a preprocessing time required for the preprocessing of the reconstruction by the server using as input the number of captured images, the resolution of the captured images, and the available computational resources of the server, or a regression model that regression-predicts the preprocessing time by the server using as input the number of captured images, the resolution of the captured images, and the available computational resources of the server, and a prediction unit that, after imaging by the moving object, inputs the number of captured images that capture the partial region, the resolution of the captured images, and a default allocation of the computational resources of the server into the learning model or the regression model, and predicts the result output by the learning model or the regression model as the preprocessing time, and the determination unit may determine the 3D reconstruction plan, including at least one of determining the processing priority of the server and allocating the computational resources of the server, for each of the partial regions associated with each of the execution units, further based on the predicted preprocessing time, on the condition that the delivery date is met.

[0019] A second aspect of the present invention is an information processing method, in which a processor receives basic requirements including an imaging target area that is an area to be imaged by a moving object, a schedule including a period during which the moving object performs imaging and an interval at which the moving object repeats imaging within the period, and service level requirements including a delivery time for providing a three-dimensional structure reconstructed from images captured by the moving object and quality related to positional accuracy of the reconstructed three-dimensional structure, generates an acquisition plan including a path of the moving object on the condition that the quality is satisfied for the basic requirements, and calculates a service level requirement corresponding to each basic requirement for each imaging execution unit defined in the schedule of a plurality of the basic requirements. On the condition that each delivery date included therein is met, the step of determining a three-dimensional reconstruction plan for each of the execution units, including the acquisition plan and at least one of the start time of imaging of the moving body, the transmission time period of the acquired image captured by the moving body, and the determination of the processing priority of the server and the allocation of the server's computational resources, is executed based on (1) each of the acquisition plans, (2) movement constraints including at least one of the time periods and areas in which the moving body can move, which are set in each of the imaging target areas, and (3) at least one of the network resources of the hangar that stores the moving body and the computational resources of the server that reconstructs the three-dimensional structure.

[0020] A third aspect of the present invention is a program, which includes a function of receiving, on a computer, basic requirements including an imaging target area that is an area to be imaged by a moving body, a schedule including a period during which the moving body performs imaging and an interval at which the moving body repeats imaging within that period, and service level requirements including a delivery time for providing a three-dimensional structure reconstructed from images captured by the moving body and quality related to the positional accuracy of the three-dimensional structure after reconstruction, a function of generating an acquisition plan including a path of the moving body on the condition that the quality is satisfied, and a function of generating, for each of the imaging execution units defined in the schedule of a plurality of the basic requirements, service level requirements including quality related to the positional accuracy of the three-dimensional structure after reconstruction. The system realizes a function of determining a three-dimensional reconstruction plan for each of the execution units, including the acquisition plan and at least one of the start time of imaging of the moving body, the time period for transmitting the acquired images captured by the moving body, and the determination of the processing priority of the server and the allocation of the computational resources of the server, based on (1) each of the acquisition plans, (2) movement constraints including at least one of the time periods and areas in which the moving body can move, which are set in each of the imaging target areas, and (3) at least one of the network resources of the hangar that stores the moving body and the computational resources of the server that reconstructs the three-dimensional structure, on the condition that each delivery date set forth in the system is met.

[0021] In order to provide this program or to update a part of the program, a computer-readable recording medium on which this program is recorded may be provided, or this program may be transmitted over a communication line.

[0022] Any combination of the above components, and any transformation of the present invention into a method, device, system, computer program, data structure, recording medium, etc., are also valid aspects of the present invention. [Effects of the Invention]

[0023] According to the present invention, the purpose of utilizing mobile objects can be achieved. [Brief explanation of the drawings]

[0024] [Figure 1] FIG. 2 is a schematic diagram illustrating an overview of a process executed by an information processing device according to an embodiment. [Figure 2] FIG. 1 is a diagram schematically illustrating a functional configuration of an information processing device according to an embodiment. [Figure 3] 10 is a diagram illustrating a data structure of information relating to basic requirements and service level requirements received by a reception unit. FIG. [Figure 4] FIG. 2 is a diagram schematically illustrating a data structure of acquisition plan information for managing acquisition plans. [Figure 5] FIG. 10 is a diagram illustrating a data structure of movement constraint information that manages movement constraints. [Figure 6] FIG. 2 is a diagram schematically showing a data structure of three-dimensional reconstruction plan information that manages a three-dimensional reconstruction plan. [Figure 7] FIG. 10 is a schematic diagram for explaining a process in which a generating unit corrects an acquisition plan. [Figure 8] 10 is a diagram schematically illustrating a data structure of network resource information that manages network resources referenced by a determination unit. FIG. [Figure 9] FIG. 10 is a diagram illustrating a transition of the bandwidth utilization rate of network resources for each communication network. [Figure 10] FIG. 10 is a diagram schematically illustrating the transition of the usage of the computing resources of a server. [Figure 11] FIG. 2 is a diagram schematically showing a data structure of three-dimensional reconstruction plan information that manages a three-dimensional reconstruction plan. [Figure 12] FIG. 2 is a diagram schematically illustrating a data structure of coding profile information for managing coding profiles. [Figure 13] 10A and 10B are schematic diagrams for explaining a process in which a determination unit corrects a three-dimensional reconstruction plan. [Figure 14] FIG. 10 is a schematic diagram for explaining a partial region. [Figure 15]10 is a flowchart illustrating a flow of information processing executed by an information processing device according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0025] <Outline of the embodiment> An information processing device according to an embodiment is used to provide a customer with a three-dimensional structure reconstructed by a server from an image captured by a moving object. Here, the three-dimensional structure may be, for example, a three-dimensional point cloud or a three-dimensional mesh generated by Structure from Motion (SfM). The three-dimensional structure may also be expressed in a format using Neural Radiance Fields (NeRF) or 3D Gaussian Splatting.

[0026] For example, multiple customers may specify basic requirements, including an imaging target area to be imaged by a mobile object and a schedule for the mobile object to perform imaging, for purposes such as surveying. The multiple customers may then specify service level requirements, including the quality and delivery date of the reconstructed three-dimensional structure related to the specified basic requirements. An information processing device according to an embodiment is used to provide each customer with a three-dimensional structure for each imaging target area reconstructed by a server so as to satisfy the specified multiple basic requirements and service level requirements. Here, the information processing device may be used by the customer, or by a provider who has received a request from a customer to provide a three-dimensional structure.

[0027] 1 is a schematic diagram for explaining an overview of processing executed by an information processing device 1 according to an embodiment. The information processing device 1 according to the embodiment constitutes a part of a three-dimensional structure reconstruction system T. The three-dimensional structure reconstruction system T includes the information processing device 1, a mobile object 2, a server 3, and a hangar B. The three-dimensional structure reconstruction system T may also include other terminals, devices, etc.

[0028] The information processing device 1 is a device that controls the moving body 2 and the server 3 and manages the reconstruction of a three-dimensional structure. The information processing device 1 is, for example, a flight management system that manages the flight of a drone. The information processing device 1 is not limited to a flight management system, and may also be a tablet terminal, a personal computer, a single-board computer, a drone dock that is a base for charging drones and performing data communication, etc.

[0029] The moving body 2 is, for example, a drone, and includes a camera that captures an image of the imaging target area A and a storage device that stores the captured image. The moving body 2 is an unmanned vehicle such as a UAV (Unmanned Aerial Vehicle), a UGV (Unmanned Ground Vehicle), or a USV (Unmanned Surface Vehicle).

[0030] The server 3 is a device that reconstructs a three-dimensional structure from images captured by the moving object 2. The server 3 is a single computer or multiple computers. The server 3 may also be one or multiple virtual servers that operate on a cloud, which is a collection of computer resources.

[0031] Hangar B is a building for storing the mobile object 2, and is equipped with, for example, facilities for charging the mobile object 2 and network resources such as a communication network for communicating with the server 3. Here, the communication network is, for example, a cellular network such as 5G (5th Generation), a satellite communication network, a wired communication network, or a wireless LAN (Wi-Fi (registered trademark)). Hangar B may also be equipped with a plurality of these communication networks. Hangar B may also store the information processing device 1 and the server 3.

[0032] 1 shows an example in which an information processing device 1 receives requests from multiple customers (not shown) to provide three-dimensional structures for various areas of a hangar B installed in a coastal area in order to more effectively utilize the mobile object 2 stored in the hangar B. In the example shown in FIG. 1, the mobile object 2 is a drone.

[0033] Hereinafter, with reference to FIG. 1, an outline of the processes executed by the information processing device 1 according to the embodiment will be explained in the order of (1) to (3), and these numbers correspond to (1) to (3) in FIG.

[0034] (1) The information processing device 1 receives, for example, a first imaging target area A1, which is the westernmost area of ​​a coastal area measuring 1 km square, as the imaging target area A to be imaged by the mobile object 2, a first schedule S1 as the schedule S for the mobile object 2 to perform imaging, a first delivery date D1 as the delivery date D of the three-dimensional structure, and a first quality Q1 as the quality Q of the three-dimensional structure. The first imaging target area A1 and the first schedule S1 are included in the first basic requirement BR1, and the first delivery date D1 and the first quality Q1 are included in the first service level requirement SR1.

[0035] Here, the information processing device 1 accepts a plurality of basic requirements BR and service level requirements SR in addition to the first basic requirement BR1 and the first service level requirement SR1. Furthermore, the information processing device 1 may accept one or more corresponding service level requirements SR for one basic requirement BR.

[0036] Hereinafter, in this specification, the multiple imaging target areas A, schedule S, delivery date D, and quality Q will be simply referred to as imaging target area A, schedule S, delivery date D, and quality Q, respectively, unless otherwise specified. When distinguished, they will be referred to as, for example, first imaging target area A1, first schedule S1, first delivery date D1, and first quality Q1. Note that the second and subsequent imaging target areas will be referred to as, for example, second imaging target area A2, third imaging target area A3, and so on. Furthermore, schedule S includes the period during which the mobile object 2 performs imaging and the interval within that period at which the mobile object 2 repeats imaging, and one or more imaging execution units are defined. Here, an execution unit refers to a group of imaging performed to provide a three-dimensional structure.

[0037] (2) The information processing device 1 generates a first acquisition plan CP1 including a path of the moving body 2 for capturing an image of the first imaging target area A1, on the condition that the first quality Q1 is satisfied. The information processing device 1 similarly generates acquisition plans CP for each of the other basic requirements BR and service level requirements SR. Hereinafter, in this specification, multiple acquisition plans CP will be simply referred to as acquisition plans CP when no particular distinction is made, and when a distinction is made, they will be referred to as, for example, the first acquisition plan CP1, the second acquisition plan CP2, etc.

[0038] (3) For each of the execution units of imaging defined in each schedule S, the information processing device 1 determines, on the condition that the corresponding delivery dates D are met, an acquisition plan CP, a three-dimensional reconstruction plan TP for each execution unit, including the acquisition plan CP, the imaging start time of the moving body 2, the transmission time of the acquired image captured by the moving body 2 to the server 3, and at least one of determining the processing priority of the server 3 and allocating the computational resources of the server 3, for the multiple basic requirements BR. When determining the three-dimensional reconstruction plan TP for each execution unit, the information processing device 1 determines it based on (1) each acquisition plan CP, (2) movement constraints including at least one of the time periods and areas in which the moving body 2 can move, which are set in each imaging target area A, and (3) at least one of the network resources of the hangar B that stores the moving body 2 and the computational resources of the server 3 that reconstructs the three-dimensional structure.

[0039] Hereinafter, in this specification, the multiple three-dimensional reconstruction plans TP will be simply referred to as three-dimensional reconstruction plans TP when no particular distinction is made, and when a distinction is made, they will be referred to as, for example, a first three-dimensional reconstruction plan TP1, a second three-dimensional reconstruction plan TP2, etc. For example, for the first basic requirement BR1, when there are three imaging execution units defined in the first schedule S1, the information processing device 1 determines three three-dimensional reconstruction plans TP, from the first three-dimensional reconstruction plan TP1 to the third three-dimensional reconstruction plan TP3.

[0040] In this way, the information processing device 1 according to the embodiment determines each three-dimensional reconstruction plan TP by taking into consideration a plurality of basic requirements BR and service level requirements SR, as well as the movement constraints of the moving body 2 and resource constraints such as network resources and the computational resources of the server 3, and therefore can achieve the utilization purpose of the moving body 2, which is to provide each three-dimensional structure while satisfying each quality Q and each delivery date D.

[0041] <Functional configuration of information processing device 1 according to the embodiment> FIG. 2 is a diagram schematically illustrating the functional configuration of an information processing device 1 according to an embodiment. The information processing device 1 includes a storage unit 10, a communication unit 11, and a control unit 12. In FIG. 2, arrows indicate main data flows, and there may be data flows not shown in FIG. 2. In FIG. 2, each functional block indicates a configuration in functional units, rather than a configuration in hardware (device) units. Therefore, the functional blocks shown in FIG. 2 may be implemented in a single device, or may be implemented separately in multiple devices. Data may be exchanged between functional blocks via any means, such as a data bus, a network, or a portable storage medium.

[0042] The storage unit 10 is a large-capacity storage device such as a ROM (Read Only Memory) that stores the BIOS (Basic Input Output System) of the computer that realizes the information processing device 1, a RAM (Random Access Memory) that serves as the working area of ​​the information processing device 1, an HDD (Hard Disk Drive) or an SSD (Solid State Drive) that stores the OS (Operating System), application programs, and various information referenced when the application programs are executed.

[0043] The communication unit 11 is a communication interface for the information processing device 1 to communicate with external devices, and is realized by a known communication module such as a LAN (Local Area Network) module or a Wi-Fi (registered trademark) module. Hereinafter, in this specification, description of the communication unit 11 may be omitted on the assumption that communication between the information processing device 1 and external devices is via the communication unit 11.

[0044] The control unit 12 is a processor such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), or NPU (Neural network Processing Unit) of the information processing device 1, and functions as a reception unit 121, a generation unit 122, a determination unit 123, a difference acquisition unit 124, an encoding unit 125, a transmission unit 126, a prediction unit 127, and a reconstruction startup unit 128 by executing a program stored in the memory unit 10.

[0045] 2 shows an example in which the information processing device 1 is configured as a single device. However, the information processing device 1 may be realized by a plurality of processors, memories, and other computing resources, such as a cloud computing system. In this case, each unit constituting the control unit 12 is realized by at least one of a plurality of different processors executing a program.

[0046] The reception unit 121 receives basic requirements BR including an imaging target area A, which is the area to be imaged by the moving body 2, and a schedule S including a period during which the moving body 2 will perform imaging and an interval at which the moving body 2 will repeat imaging within that period, as well as service level requirements SR including a delivery date D for providing a three-dimensional structure reconstructed from the captured images captured by the moving body 2 and a quality Q related to the positional accuracy of the three-dimensional structure after reconstruction.

[0047] Fig. 3 is a diagram schematically showing the data structure of information relating to basic requirements BR and service level requirements SR received by the reception unit 121. Fig. 3(a) is a diagram schematically showing the data structure of basic requirement information 100 that manages basic requirements BR received by the reception unit 121. Fig. 3(b) is a diagram schematically showing the data structure of service level requirement information 101 that manages service level requirements SR received by the reception unit 121.

[0048] In the example shown in Figure 3(a), the basic requirement information 100 is information that associates a ``basic requirement identifier'' for identifying the basic requirement BR, a ``target area identifier'' for identifying the target area A, ``target area information'' that indicates the range of the geographical area occupied by the target area A, and ``first imaging date and time,'' ``last imaging date and time,'' and ``imaging frequency'' that are information that indicates the schedule S.

[0049] In the example shown in FIG. 3(a), the "imaging target area information" is information in a polygon format in which the range of the geographical area occupied by the imaging target area A is expressed by a plurality of vertex coordinates. The imaging target area A accepted by the accepting unit 121 may be a structure such as a steel tower, or a building or other structure. The imaging target area A accepted by the accepting unit 121 may also be an area within a specified area where a landslide has occurred. The "imaging area" is not limited to a polygon format, and may also be an address, the name of a natural object, the name of a building, or the like.

[0050] In the example shown in Figure 3(b), the service level requirement information 101 is information that associates a "basic requirement identifier" for identifying a basic requirement BR associated with a service level requirement SR, a "service level requirement identifier" for identifying a service level requirement SR, an "imaging date" that is information indicating the date on which the mobile body 2 will perform imaging, a "delivery date" that is information indicating the delivery date D, and a "quality target," "quality index name," and "quality value" that are information indicating the quality Q.

[0051] 3(b), there is service level requirement information 101 corresponding to each date on which the mobile object 2 performs imaging. However, the format is not limited to this, and for example, if the corresponding basic requirements BR, delivery date D, and quality Q are the same, the information processing device 1 may collectively manage these as the same service level requirement information 101. In this case, for example, the "imaging date" is expressed in a format such as "2025 / 10 / 1, 2025 / 10 / 2, . . . ", and the "delivery date" is expressed in a format such as "21:00 on the same day as the imaging date."

[0052] In the example shown in Fig. 3(b), quality Q is expressed by a "quality object" that indicates the object or object information to be evaluated for quality Q, a "quality index name" that indicates the standard or scale for evaluating quality Q, and a "quality value" that indicates the specific numerical value or grade required as a quality index. In the example shown in Fig. 3(b), service level requirement information 101 with a service level requirement identifier of "SRID001" has a quality object of three-dimensional structure, a quality index name of positional accuracy, and a quality value of 5 cm or less.

[0053] As another example, the quality Q may be a quality where the quality target is an acquired image, the quality index name is ground sample distance (GSD), which is an index indicating how much of the Earth's surface one pixel of the image represents, and the quality value is 3 cm or less. Note that the ground sample distance of the acquired image is a factor that limits the positional accuracy of the three-dimensional structure, and the use of images with higher ground sample distances can improve the positional accuracy of the three-dimensional structure. Instead of the quality regarding the positional accuracy of the three-dimensional structure, the quality Q may also use a quality related to the positional accuracy of the three-dimensional structure, such as the ground sample distance of the acquired image. Furthermore, the quality Q may also include a quality unrelated to the positional accuracy of the three-dimensional structure.

[0054] The generation unit 122 generates an acquisition plan CP including a path for the moving object 2 on the condition that the quality Q is satisfied for the basic requirement BR. For example, the generation unit 122 calculates captured images that should be acquired in order to satisfy the quality Q from the quality Q and the area and shape of the imaging target area A, and generates an acquisition plan CP including a path for capturing the captured images. The generation unit 122 may generate a path for capturing the minimum number of captured images required to satisfy the quality Q, or may generate a path for capturing more images than the minimum number of captured images required to increase the quality Q of the three-dimensional structure.

[0055] 4 is a diagram schematically illustrating the data structure of acquisition plan information 102 that manages the acquisition plan CP. In the example illustrated in FIG. 4, the acquisition plan information 102 is information in which an "acquisition plan identifier" for identifying the acquisition plan CP, a "service level requirement identifier" for identifying the service level requirement SR that triggers the generation of the acquisition plan CP, a "travel route" that is information indicating the travel route of the moving object 2, a "travel time required" that indicates the time required to travel along the travel route, an "imaging route" that is information indicating a route of the travel route along which imaging is performed, a "imaging time required" that indicates the time required to image along the imaging route, and an "imaging density" are associated with each other. The "imaging density" will be described later.

[0056] The determination unit 123 determines, for each execution unit of imaging defined in the schedule S of a plurality of basic requirements BR, a three-dimensional reconstruction plan TP for each execution unit, on the condition that each delivery date D included in the service level requirement SR corresponding to each basic requirement BR is satisfied, the three-dimensional reconstruction plan TP for each execution unit including an acquisition plan CP, an imaging start time of the moving body 2, a time period for transmitting the acquired image captured by the moving body 2 to the server 3, and at least one of determination of the processing priority of the server 3 and allocation of computational resources of the server 3. When determining the three-dimensional reconstruction plan TP for each execution unit, the determination unit 123 determines it based on (1) each acquisition plan CP, (2) movement constraints including at least one of a time period and a movement area in which the moving body 2 can move, which are set in each imaging target area A, and (3) at least one of network resources of a hangar B that stores the moving body 2 and computational resources of the server 3 that reconstructs the three-dimensional structure.

[0057] The determination unit 123 determines the three-dimensional reconstruction plan TP for each execution unit so as to satisfy, for example, the conditions that each delivery date D is met, that each acquisition plan CP is executed, that the mobile body 2 executing each acquisition plan CP adheres to the movable time period and movable area, that the network resources required to execute the transmission processing in each three-dimensional reconstruction plan TP do not exceed the upper limit of the network resources possessed by the hangar B, and that the computational resources required to execute each three-dimensional reconstruction plan TP do not exceed the upper limit of the computational resources of the server 3.

[0058] Fig. 5 is a diagram schematically illustrating the data structure of movement restriction information 103 that manages movement restrictions. Fig. 5 shows an example in which the movement restriction information 103 manages time periods and areas in which movement of a moving object 2 is restricted, thereby managing movable time periods and movable areas. However, the present invention is not limited to this, and the movement restriction information 103 may also manage the movable time periods and movable areas themselves.

[0059] In the example shown in Figure 5, the movement constraint information 103 is information that associates a "image capture target area identifier" for identifying the image capture target area A in which restrictions are placed on the movement of the moving body 2, a "movement constraint identifier" that identifies the movement constraint on the movement of the moving body 2, a "movement constraint area" that is information indicating the area in which the movement of the moving body 2 is restricted, a "movement constraint period" and a "movement constraint time period" that indicate the time period in which the movement of the moving body 2 is restricted, and a "reason for restriction" that indicates the reason why the movement of the moving body 2 is restricted.

[0060] In the example shown in Fig. 5, when the information indicating the "movement restriction area" is indicated by "-", it means that the movement of the moving body 2 is restricted in all areas. Similarly, in the example shown in Fig. 5, when the information indicating the "movement restriction period" and the "movement restriction time zone" is indicated by "-", it means that the movement of the moving body 2 is restricted in all time zones.

[0061] Fig. 6 is a diagram schematically showing the data structure of three-dimensional reconstruction plan information 104 that manages the three-dimensional reconstruction plan TP. In the example shown in Fig. 6, the three-dimensional reconstruction plan information 104 is information in which a "service level requirement identifier" for identifying a service level requirement SR that is the trigger for determining the three-dimensional reconstruction plan TP, a "three-dimensional reconstruction plan identifier" for identifying the three-dimensional reconstruction plan TP, an "acquisition plan identifier" for identifying the acquisition plan CP included in the three-dimensional reconstruction plan TP, a "movement start time" indicating the movement start time of the moving object 2, an "imaging start time" indicating the imaging start time of the moving object 2, a "transmission time zone" indicating the time zone for transmitting the acquired image to the server 3, an "allocated GPU" and an "allocated memory" that are information indicating the computational resources of the server 3 allocated to the reconstruction processing, and a "priority" that is information indicating the processing priority of the server 3 for the reconstruction processing are associated with each other.

[0062] 6, the larger the number, the higher the "priority" is set. The server 3 may dynamically control whether or not to execute processes or the execution order based on, for example, the priority set in each three-dimensional reconstruction plan TP and a threshold that changes depending on the load state.

[0063] As described above, the information processing device 1 according to the embodiment determines each three-dimensional reconstruction plan TP by the determination unit 123, taking into consideration a plurality of basic requirements BR and service level requirements SR, as well as movement constraints of the moving object 2 and resource constraints such as network resources and computational resources of the server 3, and therefore can achieve the utilization purpose of the moving object 2, which is to provide each three-dimensional structure while satisfying each quality Q and each delivery date D.

[0064] The information processing device 1 may further include a plan execution unit that, in accordance with the three-dimensional reconstruction plan TP, performs at least one of performing imaging by the moving object 2, transmitting the acquired images to the server 3, setting processing priorities for the server 3, and allocating computational resources of the server 3, and issuing a reconstruction instruction to the server 3. This allows the information processing device 1 to have the plan execution unit execute everything from capturing images by the moving object 2 to reconstructing the three-dimensional structure by the server 3 in accordance with the three-dimensional reconstruction plan TP, and can provide each three-dimensional structure while satisfying each quality Q and each delivery date D.

[0065] When the determination unit 123 identifies that there is a delivery date D that cannot be met among the delivery dates D associated with each of the multiple execution units, the generation unit 122 modifies the acquisition plan CP so that the time required to execute the acquisition plan CP is shortened for the execution units associated with the delivery date D, provided that the service level requirement SR is met.

[0066] Fig. 7 is a schematic diagram for explaining the process of correcting the acquisition plan CP by the generation unit 122. Fig. 7(a) is a diagram schematically showing the acquisition plan CP generated by the generation unit 122. Fig. 7(b) is a diagram schematically showing the acquisition plan CP corrected by the generation unit 122. Fig. 7 shows an example in which a coastal area of ​​1 km square is accepted by the acceptance unit 121 as the imaging target area A, and the route included in the acquisition plan CP is represented by a dotted line.

[0067] FIG. 7( a) illustrates an example in which the route included in the acquisition plan CP generated by the generation unit 122 is a route that meanders through the imaging target area A using six straight-line sections extending in the east-west direction. In this case, it is assumed that the determination unit 123 determines that the delivery date D associated with the execution unit associated with the acquisition plan CP cannot be met. Then, the generation unit 122 modifies the acquisition plan CP so as to shorten the time required to execute the acquisition plan CP, on the condition that the related service level requirement SR is satisfied. In the example illustrated in FIG. 7( b), the route included in the acquisition plan CP modified by the generation unit 122 is a route that meanders through the imaging target area A using four straight-line sections extending in the east-west direction. This modification shortens the route traveled by the moving object 2, thereby shortening the time required to execute the acquisition plan CP.

[0068] The determination unit 123 re-determines the three-dimensional reconstruction plan TP for the execution unit using the corrected acquisition plan CP. As a result, even if the information processing device 1 cannot provide each three-dimensional structure while satisfying the basic requirements BR and the service level requirements SR using each acquisition plan CP generated by the generation unit 122, the information processing device 1 can provide each three-dimensional structure while more reliably satisfying each quality Q and each delivery date D by causing the generation unit 122 to correct the acquisition plan CP and causing the determination unit 123 to re-determine the three-dimensional reconstruction plan TP.

[0069] The determination unit 123 determines a transmission time period for an image captured by the mobile object 2 and a communication network to be used during the transmission time period, further based on the network resources for each communication network, with the further condition that the maximum amount of network resource usage per predetermined unit time for each communication network is smaller. Here, network resources refer to communication resources including a communication bandwidth that can be allocated for communication in the communication network. Examples of such communication networks include cellular networks such as 5G, satellite communication networks, wired communication networks, and wireless LANs (Wi-Fi). Here, the predetermined unit time refers to the length of time used by the determination unit 123 to evaluate the amount of network resource usage. This time period may be determined experimentally taking into account periodic fluctuations in bandwidth usage rates in the communication networks and the average period from when the mobile object 2 starts capturing images to the delivery date D, and is, for example, one day.

[0070] Fig. 8 is a diagram schematically illustrating the data structure of network resource information 105 that manages network resources referenced by determination unit 123. In the example illustrated in Fig. 8, network resource information 105 is information that associates a "dungeon identifier" for identifying a depot B that has a network resource, a "network resource identifier" for identifying the network resource, a "communication network type" that is information indicating the type of communication network that provides the network resource, including wireless LAN and 5G, a "bandwidth" that indicates the amount of data that can be transmitted per unit time, and a "unit price (yen / MB)." The "unit price (yen / MB)" will be described later.

[0071] FIG. 9 is a diagram schematically showing the transition of the bandwidth usage rate of the network resources for each communication network. FIG. 9 shows an example in which the solid line represents the bandwidth usage rate of 5G and the dotted line represents the bandwidth usage rate of wireless LAN. FIG. 9(a) is a diagram schematically showing the transition of the bandwidth usage rate of the network resources for each communication network when the determination unit 123 determines the three-dimensional reconstruction plan TP without including the condition that the maximum value of the network resource usage be smaller. FIG. 9(b) is a diagram schematically showing the transition of the bandwidth usage rate of the network resources for each communication network when the determination unit 123 determines the three-dimensional reconstruction plan TP with the additional condition that the maximum value of the network resource usage be smaller. FIG. 9(c) will be described later.

[0072] In the example shown in Figure 9(a), for example, 5G, which has a large bandwidth, is mainly used, and the transition of the bandwidth utilization rate exhibits a mountain-like change, with the maximum value of the bandwidth utilization rate being relatively high. The determination unit 123 determines the transmission time period for the acquired image to the server 3 for each execution unit and the communication network to be used during that transmission time period so as to satisfy, for example, the conditions that each delivery date D is met and that the maximum value of the network resource usage per predetermined unit time for each communication network is reduced. Figure 9(b) shows an example in which, for example, 5G and wireless LAN are used to the same extent, and the bandwidth utilization rate is leveled out.

[0073] As a result, the information processing device 1 can reduce the risk of communication delays and degradation of communication quality by reducing the maximum value of network resource usage using the decision unit 123, and can provide each three-dimensional structure while reliably satisfying each quality Q and each delivery date D.

[0074] The determination unit 123 determines the transmission time period for the acquired image captured by the mobile body 2 and the communication network to be used during that transmission time period, further based on cost information including the network resource usage cost for each communication network, with the further condition that the total network resource usage cost within a specified period is smaller.

[0075] Here, the predetermined period is a period for the determination unit 123 to evaluate the total usage cost of the network resources. This period may be experimentally determined taking into consideration the unit period for which the usage cost of a communication network is calculated by a communication carrier, and may be, for example, one month. As described with reference to FIG. 8, the network resource information 105 is information that is associated with a "unit price (yen / MB)" that indicates the usage cost of the network resource for each communication network, in addition to a network resource identifier, a communication network type, a bandwidth, etc. For example, in the example shown in FIG. 8, the unit price (yen / MB) of the network resource information 105 for which the communication network type is wireless LAN is X yen / MB, and the unit price (yen / MB) of the network resource information 105 for which the communication network type is 5G is Y yen / MB.

[0076] FIG. 9(c) is a diagram schematically illustrating the transition of the bandwidth utilization rate of the network resources for each communication network when the determination unit 123 determines the 3D reconstruction plan TP with the additional condition that the total usage cost of the network resources be reduced. FIG. 9 illustrates an example in which the solid line represents the bandwidth utilization rate of 5G, which has a relatively high unit price, and the dotted line represents the bandwidth utilization rate of wireless LAN, which has a relatively low unit price. In the example illustrated in FIG. 9(a), as described above, for example, 5G, which has a large bandwidth and a relatively high unit price, is primarily used. The determination unit 123 determines the transmission time period for the acquired image to the server 3 for each execution unit and the communication network to be used during the transmission time period, so as to satisfy, for example, the conditions that each delivery date D is met and that the total usage cost of the network resources within a predetermined period is reduced. FIG. 9(c) illustrates an example in which, for example, wireless LAN, which has a relatively low unit price, is preferentially used to the extent that the delivery date D can be met.

[0077] This allows the information processing device 1 to use a cheaper communication network through the determining unit 123, thereby reducing the cost of providing each three-dimensional structure.

[0078] The determination unit 123 determines at least one of the processing priority of the server 3 and the allocation of the computational resources of the server 3 based on the computational resources of the server 3 that reconstruct the three-dimensional structure, with the further condition that the maximum amount of computational resources used by the server 3 per specified unit time is smaller.

[0079] Here, the computational resources of the server 3 are resources available for executing processing, including, for example, a CPU, a GPU, a memory, etc. The predetermined unit time is the length of time for the determination unit 123 to evaluate the utilization of the computational resources of the server 3. This time may be determined by experiment in consideration of periodic fluctuations in the utilization rate of the computational resources of the server 3, the average time required for the server 3 to perform reconstruction processing, etc., and may be, for example, one day.

[0080] Fig. 10 is a diagram schematically showing a transition in the amount of usage of the computational resources of the server 3. Specifically, Fig. 10 is a diagram schematically showing a transition in the amount of usage of memory among the computational resources of the server 3. Fig. 10(a) is a diagram schematically showing a transition in the amount of usage of the computational resources of the server 3 when the determination unit 123 determines the three-dimensional reconstruction plan TP without including as a condition that the maximum value of the amount of usage of the computational resources of the server 3 be smaller. Fig. 10(b) is a diagram schematically showing a transition in the amount of usage of the computational resources of the server 3 when the determination unit 123 determines the three-dimensional reconstruction plan TP with the additional condition that the maximum value of the amount of usage of the computational resources of the server 3 be smaller.

[0081] In the example shown in Fig. 10(a), the transition of the usage amount shows a mountain-like change, with the maximum value of the usage amount being relatively high. For example, the determination unit 123 determines at least one of the processing priority of the server 3 and the allocation of the computing resources of the server 3 for each execution unit so as to satisfy the conditions of meeting each delivery date D and of reducing the maximum value of the usage amount of the computing resources of the server 3. Fig. 10(b) shows an example in which the usage amount of the computing resources is leveled in this way.

[0082] As a result, the information processing device 1 reduces the maximum amount of computational resources used by the determination unit 123, thereby improving the stability of the server 3 and enabling the information processing device 1 to provide each three-dimensional structure while more reliably satisfying each quality Q and each delivery date D. Furthermore, as a result, the information processing device 1 reduces the maximum amount of computational resources used by the determination unit 123, which may enable the application of a cheaper fee system for the usage fees of the computational resources, thereby potentially reducing the cost of providing each three-dimensional structure.

[0083] The determination unit 123 uses at least one of the maximum value Mn of the network resource usage amount per predetermined unit time, the maximum value Mc of the server 3's computational resource usage amount per predetermined unit time, the compensation cost P to the customer for not satisfying the service level requirement SR, the total execution cost E of each of the three-dimensional reconstruction plans TP, the variance Vd of the margin indicated by the difference between the actual delivery result and the delivery date D for each imaging target area A, and the variance Vq of the margin indicated by the difference between the actual value of the quality Q for each imaging target area A and the quality Q included in the service level requirement SR as an objective function, and uses the service level requirement SR, the available travel time slot, the available travel area, the upper limit of the network resources for each communication network, the unit usage price of the network resources for each communication network, and the upper limit of the computational resources of the server 3 as constraints, and determines the three-dimensional reconstruction plan TP by solving an optimization problem so as to minimize the objective function for each execution unit defined in the schedule S of the basic requirement BR.

[0084] Here, the objective function F is expressed as, for example, F=αMn+βMc+γP+δE+εVd+ζVq. The coefficients α, β, γ, δ, ε, and ζ are real numbers between 0 and 1, and satisfy the condition α+β+γ+δ+ε+ζ>0. Here, the optimization problem can be solved by known mixed integer optimization methods, constraint satisfaction methods, metaheuristics, heuristics using reinforcement learning, etc.

[0085] As a result, the information processing device 1 determines each three-dimensional reconstruction plan TP as an optimization problem by the determination unit 123, thereby being able to provide each three-dimensional structure while reliably satisfying each quality Q and each delivery date D, and also being able to stabilize plan creation and reduce costs.

[0086] The determination unit 123 determines a three-dimensional reconstruction plan TP that further includes the allocation of moving bodies 2 to be used in the acquisition plan CP, the time periods during which the moving bodies 2 charge, and the time periods during which the moving bodies 2 replace their batteries, based on the number of moving bodies 2 available for imaging, the number of moving bodies 2 that can be charged simultaneously, the number of available batteries, and the number of moving bodies 2 that can be stored in hangar B, with the additional condition that the distance between two different moving bodies 2 when the moving bodies 2 depart or arrive from hangar B is greater than a predetermined distance.

[0087] Here, the predetermined distance is the distance between two different moving bodies 2 to prevent collision between them. This distance may be determined experimentally taking into consideration the size of the moving bodies 2, the speed at which they take off and land, the environment at which they take off and land, etc., and is, for example, about 10 m. Furthermore, battery replacement refers to, for example, battery hot swapping, in which a battery in use is removed and replaced with a new battery without stopping the moving body 2.

[0088] Fig. 11 is a diagram schematically showing the data structure of three-dimensional reconstruction plan information 104 that manages the three-dimensional reconstruction plan TP. In the example shown in Fig. 11, the "imaging start time," "transmission time slot," "assigned GPU," "assigned memory," and "priority" described with reference to Fig. 6 are omitted and are not shown. In the example shown in Fig. 11, the three-dimensional reconstruction plan information 104 is information that associates, in addition to the information described with reference to Fig. 6, an "identifier" of an "assigned mobile object," which is information indicating the allocation of a mobile object 2 used in the acquisition plan CP, a "charging time slot" of the "assigned mobile object," which indicates a time slot during which the mobile object 2 is charged, and a "battery replacement time slot" of the "assigned mobile object," which indicates a time slot during which the mobile object 2 is to replace its battery.

[0089] As a result, even when a plurality of moving bodies 2 are present, the information processing device 1 can provide each three-dimensional structure while ensuring safety and more reliably satisfying each quality Q and each delivery date D.

[0090] The difference acquisition unit 124 acquires difference image data from an acquired image captured in accordance with the acquisition plan CP included in the three-dimensional reconstruction plan TP and an image reconstructed in relation to a past execution unit (hereinafter referred to as a "past reconstructed image").

[0091] Here, the difference acquisition unit 124 may regenerate the previously reconstructed image from the reconstructed three-dimensional structure. Specifically, the difference acquisition unit 124 may regenerate an image from the same or nearby viewpoint as the previously reconstructed image from a three-dimensional structure expressed by, for example, NeRF or 3D Gaussian Splatting.

[0092] The difference acquisition unit 124, for example, associates each acquired image captured according to the acquisition plan CP with a previous reconstructed image. The difference acquisition unit 124 determines the correspondence based on, for example, information about the path included in the acquisition plan CP, information about the imaging time, and the like. The difference acquisition unit 124 may associate two or more previous reconstructed images with one acquired image. The difference acquisition unit 124 calculates difference data between each acquired image and the associated previous reconstructed image, and acquires the difference image data. At this time, the difference acquisition unit 124 records incidental information related to the acquisition of the difference image data, including, for example, information about the associated previous reconstructed image.

[0093] The transmission unit 126 transmits the differential image data to the server 3. The transmission unit 126 transmits, for example, additional information in addition to the differential image data. The server 3 restores each acquired image, for example, using the transmitted differential data and additional information and the previously reconstructed image. Note that, for example, after completing the execution of the acquisition plan CP, the mobile object 2 may move to hangar B, and in hangar B, the mobile object 2 may be directly connected to the information processing device 1 via a wired cable. By connecting in this manner, the information processing device 1 can treat the storage device of the mobile object 2 as its own storage medium, and can acquire differential image data from the acquired images using the difference acquisition unit 124 and the transmission unit 126 and transmit the data to the server 3.

[0094] In this way, the information processing device 1 can transmit the acquired image to the server 3 by transmitting differential image data and additional information of the acquired image, instead of transmitting the acquired image itself, using the difference acquisition unit 124 and the transmission unit 126. Generally, the data volume of the differential image data and additional information is smaller than the data volume of the image itself. Therefore, the information processing device 1 can reduce the amount of data transmitted to the server 3 using the difference acquisition unit 124 and the transmission unit 126, thereby shortening the time required for transmission, thereby more reliably providing each three-dimensional structure while satisfying each quality Q and each delivery date D.

[0095] Before generating the difference image data, the difference acquisition unit 124 performs at least one of a geometric matching process, which geometrically aligns the images for generating the difference image data based on a common coordinate system, and an optical matching process, which corrects differences in exposure or white balance. For example, after associating the acquired image with a previously reconstructed image, the difference acquisition unit 124 performs at least one of a geometric matching process and an optical matching process on at least one of the acquired image and the previously reconstructed image. At this time, the difference acquisition unit 124 records, for example, the types of the geometric matching process and the optical matching process performed and parameter information used, in the accompanying information.

[0096] Generally, when differential image data is generated after geometric matching or optical matching is performed on the image from which the differential image data is generated, the amount of data of the differential image data and the additional information becomes smaller. Therefore, the information processing device 1 can further reduce the amount of data transmitted to the server 3 by the difference acquisition unit 124 and the transmission unit 126, and shorten the time required for transmission, thereby more reliably providing each three-dimensional structure while satisfying each quality Q and each delivery date D.

[0097] The encoding unit 125 may encode the acquired image using, from among a plurality of encoding profiles, an encoding profile that relatively shortens the transmission time of the acquired image captured according to the acquisition plan CP included in the three-dimensional reconstruction plan TP, on the condition that the quality Q is satisfied. The encoding unit 125 may, for example, encode the difference image data acquired by the difference acquisition unit 124, or may encode the acquired image itself.

[0098] Here, the encoding profile defines a combination of algorithms, parameters, etc. when encoding data such as images, videos, etc. Encoding profiles include standard-compliant profiles such as JPEG Baseline that conform to international standards, and unique profiles that are uniquely defined for the information processing device 1.

[0099] Fig. 12 is a diagram schematically illustrating the data structure of coding profile information 106 that manages coding profiles. In the example illustrated in Fig. 12, the coding profile information 106 is information in which a "coding profile identifier" for identifying a coding profile, a "coding profile type" indicating the type of coding profile such as a standard-compliant profile or a unique profile, a "coding profile name" that is information indicating the name of the coding profile, and "quantization" indicating the strength of quantization that is permitted to be used during coding are associated with each other. The coding profile information 106 may further be associated with known parameters, etc., used during coding. Furthermore, the encoding unit 125 may encode an acquired image by combining multiple coding profiles.

[0100] By compressing the image data by encoding the acquired image using the encoding unit 125, the amount of data to be transmitted to the server 3 can be reduced, shortening the time required for transmission. On the other hand, increasing the compression rate tends to reduce the quality Q of the three-dimensional structure reconstructed from the encoded acquired image due to information loss associated with the encoding process, so there is a trade-off between the time required for transmission and the quality Q. Furthermore, while there are encoding processes that have little information loss and a high compression rate, such encoding processes tend to require a long processing time for encoding. Therefore, the encoding unit 125 selects an encoding profile from the encoding profile information 106 that relatively shortens the sum of the time required for encoding and the time required for transmission, provided that the quality Q is satisfied, and encodes the acquired image using that encoding profile.

[0101] The transmission unit 126 may transmit the encoded acquired image to the server 3. This allows the information processing device 1 to shorten the time required for transmission while satisfying the quality Q using the encoding unit 125 and the transmission unit 126, and therefore, it is possible to provide each three-dimensional structure while more reliably satisfying each quality Q and each delivery date D.

[0102] The transmission unit 126 preferably chunks the data to be transmitted and transmits it using a resumable communication method, such as HTTP / 2 or QUIC / TLS. The transmission unit 126 may add a hash value, such as SHA-256, to verify the integrity of the data to be transmitted. The transmission unit 126 may also add forward error correction, such as ProMPEG FEC or RaptorQ, as redundancy. The transmission unit 126 may use a bandwidth control method, such as a token bucket, to control the throughput used for transmitting each 3D reconstruction plan TP that has an overlapping transmission time slot so that it is uniform. This allows the information processing device 1 to transmit the data to be transmitted more efficiently or uniformly using the transmission unit 126, thereby more reliably providing each 3D structure while satisfying each quality Q and each delivery date D.

[0103] The generation unit 122 generates an acquisition plan CP that further includes an imaging density. Here, the imaging density is an index indicating the degree of overlap between captured images, and includes a traveling direction overlap rate and a horizontal direction overlap rate. As described with reference to FIG. 4, the acquisition plan information 102 is information that is associated with the "imaging density" in addition to the travel route, imaging route, etc. For example, in the example shown in FIG. 4, in the acquisition plan information 102 whose acquisition plan identifier is CPID001, the traveling direction overlap rate and horizontal direction overlap rate included in the imaging density are 60% and 80%, respectively.

[0104] For each 3D reconstruction plan TP, the prediction unit 127 predicts the remaining time by subtracting the total time predicted to be required to execute unfinished processes among the execution of the acquisition plan CP by the mobile body 2, the transmission processing of the acquired images, and the reconstruction processing by the server 3 from the time until the delivery date D. The prediction unit 127 predicts the time required to execute unfinished plans of the acquisition plan CP, for example, from the route and imaging density included in the acquisition plan CP and the performance of the mobile body 2. The prediction unit 127 also predicts the time required to execute unfinished processes of the transmission processing, for example, from the number of acquired images captured by the mobile body 2 or the number of captured images to be acquired, the data size of the images, and the throughput of the network resources used for transmission. The prediction unit 127 then predicts the time required to execute unfinished processes of the reconstruction processing, for example, from the number of acquired images captured by the mobile body 2 or the number of captured images to be acquired, the image resolution, and the allocated computational resources of the server 3.

[0105] When the prediction unit 127 identifies that there is a three-dimensional reconstruction plan TP whose remaining time is less than a predetermined threshold, the determination unit 123 performs at least one of the following, on the condition that the service level requirement SR is satisfied: (1) instructing the generation unit 122 to make a modification to reduce the number of captured images to be acquired in the acquisition plan CP included in the identified three-dimensional reconstruction plan TP, or a modification to shorten the time required to execute the acquisition plan CP; and (2) modifying the three-dimensional reconstruction plan TP whose transmission time zone overlaps with the transmission time zone of the identified three-dimensional reconstruction plan TP, so that the overlap of the transmission time zones becomes smaller.

[0106] Here, the predetermined threshold value is the ratio of the remaining time to the time until the delivery date D, which is used by the determination unit 123 to determine whether to modify at least one of the acquisition plan CP and the three-dimensional reconstruction plan TP. This ratio may be determined by experiment in consideration of the area and shape of the imaging target region A, the number of captured images captured by the moving body 2 or the number of captured images to be captured, the upper limit of the network resources possessed by the hangar B, the upper limit of the calculation resources of the server 3, etc., and is, for example, 0.7.

[0107] When the prediction unit 127 identifies that there is a three-dimensional reconstruction plan TP whose remaining time is less than a predetermined threshold, the determination unit 123 instructs the generation unit 122 to make at least one of a modification to reduce the number of captured images to be acquired in the acquisition plan CP included in the identified three-dimensional reconstruction plan TP and a modification to shorten the time required to execute the acquisition plan CP, on the condition that the service level requirement SR is satisfied, for example.

[0108] The generation unit 122 may, for example, reduce the number of captured images to be acquired by making a correction to lower the imaging density included in the acquisition plan CP, thereby shortening the time required to execute the acquisition plan CP. Furthermore, the generation unit 122 may, for example, change the route included in the acquisition plan CP to a route that shortens the time required for travel by taking into account the influence of wind. In this way, the generation unit 122 may shorten the time required to execute the acquisition plan CP.

[0109] Furthermore, when the prediction unit 127 identifies that there is a three-dimensional reconstruction plan TP whose remaining time is less than a predetermined threshold, the determination unit 123 modifies the three-dimensional reconstruction plan TP whose transmission time zone overlaps with the transmission time zone of the identified three-dimensional reconstruction plan TP, for example, on the condition that the service level requirement SR is satisfied, so as to reduce the overlap of the transmission time zones.

[0110] Fig. 13 is a schematic diagram for explaining the processing of correcting the three-dimensional reconstruction plan TP by the determination unit 123. Fig. 13(a) is a diagram schematically showing the transmission time zone of the three-dimensional reconstruction plan TP determined by the determination unit 123. Fig. 13(b) is a diagram schematically showing the transmission time zone of the three-dimensional reconstruction plan TP corrected by the determination unit 123. In the example shown in Fig. 13, the transmission time zones of the first three-dimensional reconstruction plan TP1, the second three-dimensional reconstruction plan TP2, and the third three-dimensional reconstruction plan TP3 are represented by solid lines.

[0111] 13(a), the transmission time zone of the first three-dimensional reconstruction plan TP1 overlaps with the transmission time zones of the second three-dimensional reconstruction plan TP2 and the third three-dimensional reconstruction plan TP3. In this case, it is assumed that the prediction unit 127 determines that the remaining time of the first three-dimensional reconstruction plan TP1 is smaller than a predetermined threshold. In this case, the determination unit 123, on the condition that the service level requirement SR is satisfied, modifies the transmission time zones of the second three-dimensional reconstruction plan TP2 and the third three-dimensional reconstruction plan TP3 so that the overlap with the transmission time zone of the first three-dimensional reconstruction plan TP1 becomes smaller, as shown in the example of FIG. 13(b).

[0112] As a result, when the remaining time is smaller than a predetermined threshold, the information processing device 1 can modify at least one of the acquisition plan CP and the three-dimensional reconstruction plan TP by the decision unit 123 so that the remaining time is extended, thereby making it possible to provide each three-dimensional structure while more reliably satisfying each quality Q and each delivery date D.

[0113] Furthermore, when the prediction unit 127 identifies that there is a three-dimensional reconstruction plan TP whose remaining time is less than a predetermined threshold, the determination unit 123 may modify the transmission time zone of the identified three-dimensional reconstruction plan TP so that there is less overlap with the transmission time zone of a three-dimensional reconstruction plan TP different from the identified three-dimensional reconstruction plan TP, on the condition that the service level requirement SR is satisfied.

[0114] As a result, the information processing device 1 can modify the three-dimensional reconstruction plan TP by the decision unit 123 so that the remaining time is extended, thereby making it possible to provide each three-dimensional structure while more reliably satisfying each quality Q and each delivery date D.

[0115] When the prediction unit 127 identifies that there is a three-dimensional reconstruction plan TP whose remaining time is greater than a predetermined threshold, the determination unit 123 may instruct the generation unit 122 to make at least one of a modification to increase the number of captured images to be acquired in the acquisition plan CP included in the identified three-dimensional reconstruction plan TP and a modification to increase the time required to execute the acquisition plan CP, on the condition that the service level requirement SR is satisfied.

[0116] Here, the predetermined threshold value is the ratio of the remaining time to the time until the delivery date D, which is used by the determination unit 123 to determine whether there is enough time to execute the three-dimensional reconstruction plan TP. This ratio may be determined by experiment in consideration of the area and shape of the imaging target region A, the number of captured images captured by the moving body 2 or the number of captured images to be captured, the upper limit of the network resources of the hangar B, the upper limit of the calculation resources of the server 3, etc., and is, for example, 0.5.

[0117] This allows the quality of the three-dimensional structure to be improved if there is sufficient time until delivery date D, and increases customer satisfaction regarding the intended use of the moving body 2.

[0118] When the prediction unit 127 identifies that there is a three-dimensional reconstruction plan TP whose remaining time is greater than a predetermined threshold, the determination unit 123 may modify the identified three-dimensional reconstruction plan TP so that the overlap between the transmission time zone of the identified three-dimensional reconstruction plan TP and the transmission time zone of a three-dimensional reconstruction plan TP different from the identified three-dimensional reconstruction plan TP is reduced, on the condition that the service level requirement SR is satisfied.

[0119] Here, the predetermined threshold value is the ratio of the remaining time to the time until the delivery date D, which is used by the determination unit 123 to determine whether there is enough time to execute the three-dimensional reconstruction plan TP. This ratio may be determined by experiment in consideration of the area and shape of the imaging target region A, the number of captured images captured by the moving body 2 or the number of captured images to be captured, the upper limit of the network resources possessed by the hangar B, the upper limit of the calculation resources of the server 3, the parallel execution status of each three-dimensional reconstruction plan TP, etc., and is, for example, 0.5.

[0120] As a result, the information processing device 1 can cause the determination unit 123 to adjust the transmission time period of the three-dimensional reconstruction plan TP, which has ample time until the delivery date D, so that it does not overlap with the transmission time period of a different three-dimensional reconstruction plan TP, thereby reducing contention for network resources and more reliably providing each three-dimensional structure while satisfying each quality Q and each delivery date D.

[0121] When the prediction unit 127 identifies that there is a three-dimensional reconstruction plan TP whose remaining time is greater than a predetermined threshold, the determination unit 123 may make a modification to lower the processing priority of the server 3 included in the identified three-dimensional reconstruction plan TP, or may make a modification to reduce the allocation of computational resources of the server 3 included in the identified three-dimensional reconstruction plan TP, on the condition that the service level requirement SR is satisfied.

[0122] Here, the predetermined threshold value is the ratio of the remaining time to the time until the delivery date D, which is used by the determination unit 123 to determine whether there is enough time to execute the three-dimensional reconstruction plan TP. This ratio may be determined by experiment in consideration of the area and shape of the imaging target region A, the number of captured images captured by the moving body 2 or the number of captured images to be captured, the upper limit of the network resources possessed by the hangar B, the upper limit of the calculation resources of the server 3, the parallel execution status of each three-dimensional reconstruction plan TP, etc., and is, for example, 0.5.

[0123] As a result, the information processing device 1 can cause the determination unit 123 to adjust the computational resources of the server 3 for a three-dimensional reconstruction plan TP that has ample time until the delivery date D so that it can be used by a different three-dimensional reconstruction plan TP, thereby reducing contention for computational resources and more reliably providing each three-dimensional structure while satisfying each quality Q and each delivery date D.

[0124] The generation unit 122 divides the imaging target area A into multiple partial areas PA, and generates a route of the moving object 2 included in the acquisition plan CP as a series of routes that are capable of generating a point cloud of the partial areas PA that is sparse compared to the three-dimensional structure provided, using only images captured from the route, and is a closed-loop route with each partial area PA as an imaging target. Hereinafter, the point cloud of the partial area PA that is sparse compared to the three-dimensional structure (hereinafter referred to as a "sparse point cloud" in this specification) is, for example, a point cloud composed only of feature points of the partial area PA. Hereinafter, in this specification, the multiple partial areas PA will be simply referred to as partial areas PA when not particularly distinguished, and when distinguished, will be referred to as, for example, a first partial area PA1, a second partial area PA2, etc.

[0125] FIG. 14 is a schematic diagram for explaining the partial area PA. FIG. 14(a) is a diagram schematically illustrating the imaging target area A before it is divided into the partial areas PA. FIG. 14(b) is a diagram schematically illustrating the partial area PA and a route in which the partial area PA is the imaging target. FIG. 14(a) shows an example in which a 1 km square coastal area is accepted by the accepting unit 121 as the imaging target area A. FIG. 14(b) shows an example in which the imaging target area A is divided into seven partial areas PA, from the first partial area PA1 to the seventh partial area PA7. FIG. 14(b) shows an example in which the route in which the seventh partial area PA7 is the imaging target of the partial area PA has a concave closed shape. Here, a closed-shaped route in this specification is not limited to a route in which the start point and end point of the movement of the moving object 2 coincide, but also includes a route in which the start point and end point are relatively short distances apart.

[0126] The transmitter 126 transmits the acquired images to the server 3 in order for each partial area PA. Here, the order refers to, for example, the chronological order in which the partial area PA was imaged. The reconstruction starter 128 causes the server 3 to start, independently for each partial area PA, reconstruction preprocessing for generating a sparse point cloud for the partial area PA. Here, the reconstruction preprocessing for generating a sparse point cloud for the partial area PA includes, for example, feature point extraction for extracting feature points from multiple images included in the partial area PA, initial calibration for initially estimating and adjusting the camera position, orientation, etc. of each image, and processing for generating a sparse point cloud using the extracted feature points.

[0127] This enables the information processing device 1 to have the server 3 generate sparse point clouds for each partial area PA in parallel, thereby shortening the time required for reconstruction and more reliably providing each three-dimensional structure after meeting each delivery date D. Furthermore, the reconstruction starting unit 128 may cause the server 3 to independently start pre-processing of reconstruction for each partial area PA, which generates a sparse point cloud for the partial area PA, each time transmission of each partial area PA is completed. This allows the information processing device 1 to have the server 3 start processing for each partial area PA in advance before the transmission process is completed, and more reliably providing each three-dimensional structure after meeting each delivery date D.

[0128] The generation unit 122 may generate the acquisition plan CP, which further includes the ratio of oblique imaging, which is images captured at an angle of at least a predetermined angle relative to the vertical direction, and the imaging density. Here, the predetermined angle is an angle for acquiring the three-dimensional shape of the imaging target area A. This angle may be determined experimentally taking into account the shape of the imaging target area A, the camera parameters of the camera equipped on the moving object 2, and the characteristics of the reconstruction process, and may be, for example, 30 degrees. In this case, the generation unit 122 may generate the acquisition plan CP so that a predetermined acquisition density is ensured near the boundary of the partial area PA, and the imaging orientations of the oblique imaging images form a predetermined crossing angle with each other. This allows the information processing device 1 to improve the quality Q when integrating the sparse point clouds of the partial area PA using the generation unit 122, and to provide each three-dimensional structure while more reliably satisfying each quality Q.

[0129] Furthermore, when the prediction unit 127 identifies that there is a three-dimensional reconstruction plan TP whose remaining time is less than a predetermined threshold, the determination unit 123 may instruct the generation unit 122 to at least one of modifying the acquisition plan CP included in the identified three-dimensional reconstruction plan TP to reduce the number of captured images to be acquired or modifying the acquisition plan CP to shorten the time required to execute the acquisition plan CP, on the conditions that the service level requirement SR is satisfied and the imaging directions of the oblique images near the boundary of the partial area PA form a predetermined intersection angle with each other.

[0130] Here, the generation unit 122 may perform a modification to reduce the number of captured images to be acquired in the acquisition plan CP and a modification to shorten the time required to execute the acquisition plan CP by lowering the ratio of oblique imaging included in the acquisition plan CP. This allows the information processing device 1 to ensure the quality Q when integrating the sparse point clouds of the partial area PA by the determination unit 123, and further enables the information processing device 1 to provide each three-dimensional structure while reliably satisfying each quality Q and each delivery date D.

[0131] The storage unit 10 stores a learning model that has been trained to output a preprocessing time required for preprocessing of reconstruction by the server 3 using as input the number of captured images, the resolution of the captured images, and the computational resources available on the server 3, or a regression model that regression-predicts the preprocessing time by the server 3 using as input the number of captured images, the resolution of the captured images, and the computational resources available on the server 3. After the moving object 2 performs imaging, the prediction unit 127 inputs, for each partial area PA, the number of captured images that capture the partial area PA, the resolution of the captured images, and a default allocation of the computational resources of the server 3 into the learning model or the regression model, and predicts the result output by the learning model or the regression model as the preprocessing time.

[0132] Here, the default allocation of the computational resources of the server 3 refers to the allocation of the computational resources of the server 3 that is predetermined as an initial setting by the determination unit 123. The determination unit 123 may, for example, determine this allocation for each customer, for each basic requirement BR, or for each service level requirement SR. When the three-dimensional reconstruction plan TP does not include the allocation of the computational resources of the server 3, this default allocation may be applied.

[0133] The determination unit 123 determines a three-dimensional reconstruction plan TP including at least one of determining the processing priority of the server 3 and allocating the computing resources of the server 3, for each partial area PA associated with each execution unit, further based on the predicted preprocessing time, on the condition that the delivery date D is met.

[0134] As a result, the information processing device 1 can control at least one of determining the processing priority of the server 3 and allocating the computing resources of the server 3 on a sub-area PA basis using the determination unit 123, thereby more reliably providing each three-dimensional structure while meeting each delivery date D.

[0135] The storage unit 10 may store a learning model that has been trained to output a preprocessing time required for preprocessing of reconstruction by the server 3 using as input the number of captured images, the resolution of the captured images, the computational resources available to the server 3, and an index related to the time required for reconstruction processing, or a regression model that uses as input the number of captured images, the resolution of the captured images, the computational resources available to the server 3, and an index related to the time required for reconstruction processing. After the mobile object 2 performs imaging, the prediction unit 127 may input, for each partial area PA, the number of acquired images capturing the partial area PA, the resolution of the acquired images, a default allocation of computational resources to the server 3, and an index related to the time required for reconstruction processing, into the learning model or the regression model, and predict the result output by the learning model or the regression model as the preprocessing time.

[0136] Here, the index related to the time required for reconstruction processing is, for example, the feature point density, which is the number of feature points in the image to be reconstructed, or the difficulty of convergence of estimation in initial calibration, which is the number of iterations required until the estimated value converges to a desired accuracy in initial calibration or the difficulty of stability. This enables the information processing device 1 to improve the accuracy of prediction by the prediction unit 127, and to provide each three-dimensional structure while more reliably satisfying each quality Q and each delivery date D.

[0137] The determination unit 123 may determine, for each partial area PA associated with each execution unit, a transmission time period for an acquired image capturing each partial area PA captured by the moving object 2 and a communication network to be used during that transmission time period, further based on the network resources for each communication network, with the further condition that the maximum amount of network resource usage per predetermined unit time for each communication network is smaller. This allows the information processing device 1 to control the transmission time period and the communication network to be used during that transmission time period for each partial area PA by the determination unit 123, and therefore, can provide each three-dimensional structure while more reliably meeting each delivery date D.

[0138] <Processing flow of information processing method executed by information processing device 1> 15 is a flowchart for explaining the flow of information processing executed by the information processing device 1 according to the embodiment. The processing in this flowchart starts, for example, when the information processing device 1 is started.

[0139] The receiving unit 121 receives basic requirements BR including an imaging target area A and a schedule S, and service level requirements SR including a delivery date D and quality Q (S1). The generating unit 122 generates an acquisition plan CP including a route for the moving object 2, on the condition that the quality Q is satisfied for the basic requirements BR (S2).

[0140] The determination unit 123 determines a three-dimensional reconstruction plan TP for each execution unit of imaging defined in the schedule S of multiple basic requirements BR, based on (1) each acquisition plan CP, (2) movement constraints set for each imaging target area A, and (3) at least one of the network resources of the hangar B and the computational resources of the server 3, and includes the acquisition plan CP, the imaging start time of the moving body 2, the transmission time period for the acquired image captured by the moving body 2, and at least one of the determination of the processing priority of the server 3 and the allocation of the computational resources of the server 3, on the condition that each delivery date D included in the service level requirement SR corresponding to each basic requirement BR is satisfied (S3).

[0141] The determination unit 123 determines whether each of the three-dimensional reconstruction plans TP determined for each execution unit satisfies the corresponding delivery date D, and whether all of the delivery dates D are met (S4). If all of the delivery dates D are met (Yes in S4), the processing in this flowchart ends.

[0142] If there is a delivery date D that cannot be met (No in S4), the generation unit 122 modifies the acquisition plan CP for the execution unit associated with the delivery date D so as to shorten the time required to execute the acquisition plan CP, on the condition that the service level requirement SR is satisfied (S5). The determination unit 123 re-determines the three-dimensional reconstruction plan TP for the execution unit using the modified acquisition plan CP (S6). When the determination unit 123 has re-determined the three-dimensional reconstruction plan TP, the processing in this flowchart ends.

[0143] <Advantages of the information processing device 1 according to the embodiment> As described above, the information processing device 1 according to the embodiment can achieve the purpose of utilizing the moving object 2.

[0144] Furthermore, this invention will make it possible to contribute to Goal 9 of the United Nations' Sustainable Development Goals (SDGs), which is "Build resilient infrastructure, promote inclusive and sustainable industrialization, and promote innovation and resilience."

[0145] The present invention has been described above using embodiments, but the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. For example, all or part of the device can be configured by functionally or physically distributing or integrating any unit. Furthermore, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effects of the new embodiments resulting from the combination also have the effects of the original embodiments.

[0146] <Modification> The example according to the modified example is a specific operational form that realizes the present invention as a survey image transmission service that targets daily surveying operations at multiple locations and is centered on optimization on a location-by-day basis. The information processing device 1 according to the modified example comprehensively calls the reception unit 121, generation unit 122, determination unit 123, prediction unit 127, etc. described in the above embodiment from the service layer, and provides operation management, visualization, analysis, improvement proposals, estimates, billing, settlement, distribution, and audits in an integrated manner.

[0147] This service is provided by an information processing device 1 according to a modified example via a management portal and API (Application Programming Interface). The customer or provider's operations staff registers the imaging target area A, schedule S, quality Q, and delivery date D for each location and day, and the registered information is accepted as basic requirements BR and service level requirements SR. This service uses the accepted information as input to generate and modify plans across locations and days, and provides a workflow for issuing and approving monitoring and correction instructions during operation.

[0148] 2 , the information processing device 1 according to the modification may be configured to further include a base management unit 129, an operation performance collection unit 130, a visualization unit 131, an abnormality detection unit 132, an improvement proposal unit 133, a contract / price plan management unit 134, a billing unit 135, an API management unit 136, a fee estimation unit 137, a payment linkage unit 138, a distribution unit 139, and an audit log unit 140, all of which are not shown. The base management unit 129 manages attributes such as the base identifier, location, linkage with Hangar B, available communication network type, bandwidth, and unit price, available flight time slots, and data residency conditions. The operation performance collection unit 130 collects and aggregates key metrics related to imaging, transmission, and reconstruction (number of images, data volume, communication network used, processing requirements, delivery time, quality performance, etc.) on a base-by-base and daily basis. The visualization unit 131 displays the achievement of SLA (Service Level Agreement) by location and day, costs, trends in network and server load, transmission overlaps, etc., and generates standard reports. The anomaly detection unit 132 detects signs of cost overruns, load peaks, risk of delay in delivery date D, and quality degradation. The improvement proposal unit 133 generates corrective proposals such as rearrangement of transmission time slots, communication network switching, change of encoding profile, fine-tuning of acquisition plan CP, etc., and reflects them upon approval.

[0149] The contract / pricing plan management unit 134 manages the tiered / pay-as-you-go billing and subscription configuration based on the number of locations, daily usage volume, accuracy rank, etc., SLAs (delivery date / quality class, refund coefficient in case of breach, tiered provision conditions, etc.), unit prices (acquisition time unit price, encoded data volume unit price, transmission data volume unit price, GPU hourly unit price, storage capacity unit price), tiered fees, regional coefficients, and time zone coefficients. The billing unit 135 calculates expenses based on operational performance, applies leveling incentive discounts, calculates credits for unmet SLAs, and performs billing processing. The API management unit 136 is responsible for authentication and authorization of external collaboration and event notification. The fee estimation unit 137 calculates estimates, the payment collaboration unit 138 collaborates with payment methods such as invoices, direct debits, credit cards, bank transfers, and prepaid payments, and the distribution unit 139 distributes revenue among related businesses. The audit log unit 140 stores time-stamped meter information, model input / output, fee formula version, policy identifier, and the like, which are the basis for estimates, confirmations, refunds, and distributions.

[0150] The main data handled by the information processing device 1 relating to the modified example in this service are base information (base ID, hangar identifier, bandwidth and unit price for each type of communication network, flight hours, data residency conditions, etc.), daily SLA (base ID, imaging date, quality Q, delivery date D, priority, etc.), operational performance (number of images captured, data volume, communication network breakdown, processing requirements, delivery time, quality performance, etc.), fee plan (basic fee, number of base tier, daily usage tier, accuracy rank coefficient, communication network unit price, computational resource unit price, addition of reports, etc.), and metrics (bandwidth utilization rate, computational resource utilization rate, SLA margin, cost breakdown, anomaly score).

[0151] In daily operations, when site and daily requirements are registered the day before or the morning of the day, the information processing device 1 according to the modified example generates a plan based on the requirements and initiates cross-sectional optimization. During operation, based on the prediction and monitoring of remaining time, load, and cost, correction proposals for eliminating overlapping transmission time slots, reallocating resources, and minor adjustments to the acquisition plan CP are generated and applied upon approval by the customer or provider's operations staff. The applied corrections are immediately displayed to the customer or provider's operations staff along with their impact on SLA margins and estimated costs.

[0152] In monthly processing, operational results are compiled and costs are calculated based on the amount of data per communication network type and time period, and the use of computing resources. The contract / price plan management unit 134 applies tiered / pay-as-you-go billing according to the number of locations, daily usage volume, and accuracy rank, and the billing unit 135 determines the billing amount by reflecting the leveling incentive discount and SLA underachievement credit. A report is output that includes a breakdown of the billing details and the estimated costs for the next month.

[0153] The objective function of cross-site optimization is weighted to minimize the total communication and computing resource costs across all sites and all days, while leveling out network and server loads and suppressing penalties for non-achievement of SLAs and variance in SLA margins between sites and days. Constraints include fulfillment of each SLA, bandwidth limits for each communication network, server computing resource limits, operation policies for each site (recommended time slots, budget limits, preferred sites), data residency designations, etc. These definitions are stored as policies in the contract / pricing plan management unit 134 and are automatically updated as changes occur.

[0154] The leveling incentive discount is calculated by the accounting unit 135 using the ratio between the maximum and median (or average) utilization rates of the network and server 3 over a specified period to calculate the leveling degree, and a gradual discount rate is applied when the ratio is less than a specified threshold. An additional discount may also be applied to bases and days whose transmission plans are moved during the recommended time period. The discount application status is displayed to the customer or provider's operations staff along with the expected billing amount.

[0155] The underpayment credit for billing is calculated for each location and each day based on the degree of underpayment based on the delay time of delivery date D and the exceedance rate of quality Q, and a coefficient defined in the contract / price plan management unit 134 is applied, and the credit amount based on the cost for that day or monthly cost is calculated by the billing unit 135. The calculation basis and the expected billing amount after deductions are presented to the customer or the provider's operations staff by the visualization unit 131.

[0156] The fee estimation unit 137 applies dynamic coefficients reflecting priority, quality, and congestion when calculating the estimate. At least, the following coefficients are used: a delivery urgency coefficient Kd (which increases inversely proportional to the remaining time until the delivery date D), a quality strictness coefficient Kq (a monotonically increasing function of the required positional accuracy, the specified GSD, the oblique imaging ratio, and the imaging density), and a congestion coefficient Kl (which is proportional to the server 3's computational resource utilization rate and startup queue length). The unit price is multiplied by the tiered fee, the area coefficient, and the time-of-day coefficient to obtain the effective unit price. The estimated amount is calculated by adding the sum of the products of the basic fee and the effective unit price with the predicted values ​​of acquisition time, encoded data volume, transmission data volume, GPU time, storage capacity, etc., and then multiplying the sum by the composite coefficients Kd, Kq, and Kl. The final amount may be adjusted by applying the quality performance coefficient Kqa and the delivery date achievement coefficient Kdly.

[0157] The receiving unit 121 may receive a budget upper limit Bmax for each job. The determining unit 123 dynamically determines the parallelism, priority, and computational resource allocation that satisfies the SLA within the range of Bmax, based on the estimated required time by the predicting unit 127 and the estimate by the fee estimating unit 137. If it is expected that Bmax will be exceeded, the visualizing unit 131 presents alternatives, such as reducing the imaging density, reducing the oblique imaging ratio, changing the coding profile, or switching to staged delivery in partial area PA units, along with their impact on fees, and reflects the approval results in the determining unit 123.

[0158] This service supports gradual provision and gradual billing, with partial areas PA as processing units. Reconstruction can sequentially output intermediate results (sparse point clouds, partial meshes, etc.) for each partial area PA. When milestones defined for each partial area PA are reached, the billing unit 135 pro-ratas the variable cost portion excluding the basic fee, executes interim billing, or consumes prepaid credits, and may apply a refund coefficient based on the quality verification results. This allows the compensation for partial results and the allocation of computational resources to be synchronized, even for long jobs or in unstable communication environments.

[0159] The payment linking unit 138 links with billing, account transfer, credit card, bank transfer, prepaid credit, and inter-company payment networks. In the prepaid type, the distribution unit 139 binds the estimated amount or a certain percentage of it as transaction integrity, and distributes it sequentially to relevant parties such as the operator of the mobile object 2, the cloud service provider, and the platform provider based on distribution rules each time a milestone in the partial area PA is reached. If the SLA is not met, the payment linking unit 138 processes a refund for the unused amount or credit for the next job based on the records in the audit log unit 140 and the policy.

[0160] The audit log unit 140 stores the start and end times of each process, the device identifier, the software version, the encoding profile, the network throughput sample, the GPU allocation history, the model estimates, the intermediate and final values ​​of the fee calculation, and the version of the fee formula and policy as time-stamped records linked to the job identifier and the identifier of the partial area PA. The records are managed so that tampering can be detected using hash chains, etc., and differences between the estimated amount and the final amount can be verified by recalculating under the same input conditions, ensuring explainability in the event of an external audit or objection.

[0161] The price estimation unit 137 updates the congestion coefficient Kl in real time based on the utilization rate of the server 3, the startup waiting queue length, market price, etc., and generates price curves for the estimated stage and during execution. When surge pricing occurs, the visualization unit 131 presents proposals for changing the time zone or switching to staged delivery, along with the price impact, and the decision unit 123 updates the resource allocation depending on approval. If the customer selects cost priority, the decision unit 123 applies preemptible computing resources and pause / resume strategies to reduce total costs, and the visualization unit 131 re-presents the schedule and price to the customer, incorporating the re-execution cost and delivery date impact.

[0162] In a hybrid configuration, the contract / price plan management unit 134 internalizes the on-premise amortisation and maintenance costs as virtual unit prices, and the fee estimation unit 137 consistently presents a total cost estimate in combination with the price and congestion level on the cloud side. The determination unit 123 determines processing allocation according to security and data residency requirements, and performs optimization such as prioritising on-premise slots according to congestion and price fluctuations. The billing unit 135 issues an integrated invoice for the customer, and the distribution unit 139 automatically allocates costs and distributes revenue between the on-premise owner and the cloud provider.

[0163] In the event of a failure, the decision unit 123 performs failover to an alternative node and resets priorities, and for unattainable partial areas PA, the payment linkage unit 138 suspends staged billing and immediately refunds or credits based on the provisions of the contract / price plan management unit 134. The visualization unit 131 immediately presents expected delays, alternative plans, and fee impacts, and, upon approval, reflects the relaxation of quality Q and changes to delivery date D, and the billing unit 135 recalculates the delivery date achievement coefficient Kdly and the quality performance coefficient Kqa. The audit log unit 140 records the timeline of the failure event, impact assessment, applied compensation coefficient, and its rationale.

[0164] From the perspective of security and compliance, communications are encrypted and mutual authentication is applied as necessary. The API management unit 136 separates operational privileges (requirements registration, plan reference, performance acquisition, billing reference, policy change, etc.) based on authentication and authorization, and maintains audit logs. The base management unit 129 maintains metadata on flight times and areas and permits, as well as data residency designations, and plans are managed to comply with these constraints. The data lifecycle defines the retention period for operational performance, differential data, and encoded data, anonymization policy, and whether or not to provide to a third party, and automatic deletion or archiving is implemented.

[0165] As an external system linkage authenticated and authorized by the API management unit 136, the information processing device 1 according to the modified example is provided with the API for the registration of requirements for the day, notification of plans, acquisition of operational results and billing data, etc. Data is provided based on a machine-readable schema, and versioning and backward compatibility are maintained. When the schema is changed, transition rules and deprecation periods are defined, and advance notice and transition guidance are provided.

[0166] If necessary, preprocessing (difference generation, geometric / optical matching, feature point extraction for each partial area PA, etc.) can be applied on the edge side. The necessity of edge application and the allocation ratio are automatically determined based on the communication unit price, bandwidth status, and computing facility availability of the base maintained by the base management unit 129, as well as the cost policy maintained by the contract / price plan management unit 134. The visualization unit 131 displays the impact of the allocation on the expected cost, delay, and SLA margin, and the operations staff can adjust the policy thresholds.

[0167] In onboarding at the time of introduction, an initial predetermined period is treated as a search phase, and the operation performance collection unit 130 collects the time zone distribution of communication throughput specific to each site, cost sensitivity, tendency of calculation requirements, distribution of margin for quality Q, etc., and reflects these in subsequent cross-sectional optimization by the information processing device 1 according to the modified example. Overshoots in costs during the search phase can be adjusted as promotion credits within a range defined by the contract / price plan management unit 134.

[0168] The operation policy can be flexibly set for each tenant (customer). The overall budget limit, priority of specific locations, compliance with recommended time periods, upper limit of credits for unmet SLAs, level of detail in reports, data retention period, external collaboration method, etc. are stored in the contract / price plan management unit 134 and are reflected in the decisions of the decision unit 123 and the billing unit 135.

[0169] To ensure fairness among tenants, the information processing device 1 according to the modified example applies a resource allocation policy in a multi-tenant environment. The information processing device 1 according to the modified example allocates transmission slots and computing resources that compete in the same time period, taking into consideration the SLA priority of each tenant, the actual allocation amount for the month, and the most recent history of non-achievement. The visualization unit 131 presents the basis for resource allocation in an auditable format, and the API management unit 136 notifies the allocation results based on the policy as an event.

[0170] The billing unit 135 provides forecasts of expected costs for the current month and scenario analysis to support budget-to-actual cost management. It calculates expected communication and calculation costs based on the number of days remaining in the current month, registered daily SLAs, and past performance data, and instantly estimates the cost reduction effect assuming policy changes (strengthening shifts to recommended time slots, gradual changes in accuracy rank, etc.). The visualization unit 131 compares and presents scenarios, and updates the policy of the contract / price plan management unit 134 after approval.

[0171] The elements of the modified examples can be combined with the technical elements described in the above-described embodiments in any combination. The weights of the objective functions used in the cross-sectional optimization, the constraints, the cost parameters, the definition of the leveling degree, the credit calculation formula, the data retention policy, the authority scope, the service provision form (SaaS, on-premise, hybrid), and the processing allocation (edge / cloud) can be changed as appropriate according to the contract conditions, regulatory requirements, and operator policies, and these modifications are also included in the scope of the present invention. [Explanation of symbols]

[0172] 1. Information processing device 10...Storage section 11. Communications Department 12 Control section 121 Reception 122...Generation section 123...Decision section 124...Difference acquisition part 125...encoding section 126 Transmission unit 127···Prediction section 128...Reconfiguration startup part 2. Mobile 3. Server

Claims

1. a receiving unit that receives basic requirements including an imaging target area, which is an area that is imaged by a moving body, a schedule including a period during which the moving body performs imaging and an interval at which the moving body repeats imaging within that period, a delivery time for providing a three-dimensional structure reconstructed from the captured images taken by the moving body, and service level requirements including quality related to the positional accuracy of the three-dimensional structure after reconstruction; a generation unit that generates an acquisition plan including a route of the moving object, on the condition that the quality is satisfied for the basic requirements; a determination unit that determines, for each execution unit of imaging defined in the schedule of a plurality of the basic requirements, a three-dimensional reconstruction plan for each execution unit, including the acquisition plan, the imaging start time of the moving body, the transmission time period for the acquired images captured by the moving body, and at least one of determination of the processing priority of the server and allocation of the computational resources of the server, based on (1) each of the acquisition plans, (2) movement constraints including at least one of time periods and areas in which moving bodies set in each of the imaging target areas can move, and (3) at least one of network resources of a hangar that stores the moving bodies and computational resources of a server that reconstructs the three-dimensional structure, on the condition that each delivery date included in the service level requirements corresponding to each basic requirement is met; An information processing device comprising:

2. When the determination unit identifies that there is a delivery date that cannot be met among the delivery dates associated with each of the plurality of execution units, the generation unit modifies the acquisition plan so as to shorten the time required to execute the acquisition plan for the execution unit associated with the delivery date, on the condition that the service level requirement is met; the determination unit determines again the three-dimensional reconstruction plan for the execution unit using the corrected acquisition plan. The information processing device according to claim 1 .

3. the determination unit determines the transmission time period of the image captured by the moving body and the communication network to be used for the transmission time period further based on the network resources for each communication network, with a further condition that the maximum value of the usage amount of the network resources per predetermined unit time for each communication network is smaller. The information processing device according to claim 1 .

4. the determination unit determines the transmission time period of the acquired image captured by the moving body and the communication network to be used for the transmission time period, further based on cost information including the usage cost of the network resources for each communication network, with the further condition that the total usage cost of the network resources within a predetermined period is smaller. The information processing device according to claim 1 .

5. the determination unit determines at least one of the processing priority of the server and the allocation of the server's computing resources, with the further condition that a maximum value of the server's computing resource usage per predetermined unit time is smaller. The information processing device according to claim 1 .

6. the determination unit determines the 3D reconstruction plan by solving an optimization problem so as to minimize the objective function for each of the execution units defined in the schedule of the basic requirements, using as constraints at least one of the maximum usage amount of the network resources per predetermined unit time, the maximum usage amount of the server's computational resources per predetermined unit time, the compensation cost to a customer due to non-satisfaction of the service level requirements, the total execution cost of each of the 3D reconstruction plans, the variance of the margin indicated by the difference between the delivery record and the delivery date for each of the imaging target areas, and the variance of the margin indicated by the difference between the quality record value for each of the imaging target areas and the quality included in the service level requirements, and using as constraints the service level requirements, the movable time slot, the movable area, the upper limit of the network resources for each communication network, the unit usage price of the network resources for each communication network, and the upper limit of the server's computational resources. The information processing device according to claim 1 .

7. the determination unit determines the 3D reconstruction plan, which further includes allocation of the moving bodies to be used in the acquisition plan, time periods during which the moving bodies are charged, and time periods during which the moving bodies are to exchange batteries, based on the number of moving bodies available for imaging, the number of moving bodies that can be charged simultaneously, the number of available batteries, and the number of moving bodies that can be stored in the hangar, with the further condition that the distance between two different moving bodies is equal to or greater than a predetermined distance when the moving bodies depart from or arrive at the hangar; The information processing device according to claim 1 .

8. The information processing device includes: a difference acquisition unit that acquires difference image data from an acquired image captured according to the acquisition plan included in the three-dimensional reconstruction plan and an image reconstructed in relation to the past execution unit; a transmission unit that transmits the differential image data to the server. The information processing device according to claim 1 .

9. the difference acquisition unit performs at least one of a geometric alignment process for geometrically aligning the images for generating the difference image data based on a common coordinate system and an optical alignment process for correcting a difference in exposure or white balance, before generating the difference image data. The information processing device according to claim 8 .

10. The information processing device includes: an encoding unit that encodes the acquired image using an encoding profile that relatively shortens a transmission time of the acquired image captured according to the acquisition plan included in the three-dimensional reconstruction plan, from among a plurality of encoding profiles, on the condition that the quality is satisfied; a transmission unit that transmits the encoded acquired image to the server. The information processing device according to claim 1 .

11. The generation unit generates the acquisition plan further including an imaging density; the information processing device further includes a prediction unit that predicts a remaining time for each of the three-dimensional reconstruction plans by subtracting from the time until the delivery date a total of the time predicted to be required for executing unfinished processes among execution of the acquisition plan by the moving body, transmission processing of the acquired image, and processing of the reconstruction by the server, When the prediction unit identifies that there is a three-dimensional reconstruction plan in which the remaining time is smaller than a predetermined threshold, the determination unit performs at least one of the following, on condition that the service level requirement is satisfied: (1) instructing the generation unit to make a correction to reduce the number of captured images to be acquired in the acquisition plan included in the identified three-dimensional reconstruction plan, or a correction to shorten the time required to execute the acquisition plan; and (2) for the three-dimensional reconstruction plan whose transmission time zone overlaps with the transmission time zone of the identified three-dimensional reconstruction plan, correcting the three-dimensional reconstruction plan so that the overlap of the transmission time zones becomes smaller.

2. The information processing device according to claim 1.

12. the generation unit divides the imaging target area into a plurality of partial areas, and generates, as the route of the moving body included in the acquisition plan, a closed-loop route that targets each of the partial areas, and generates, using only images captured from the route, a series of routes from which a point cloud of the partial area that is sparse compared to the three-dimensional structure to be provided can be generated; The information processing device includes: a transmission unit that transmits the acquired images to the server in units of the partial regions in order; a reconstruction starting unit that causes a server to start a pre-processing of reconstruction for generating the point cloud of the partial region independently for each partial region, The information processing device according to claim 1 .

13. The information processing device includes: a storage unit that stores a learning model that has been trained to output a preprocessing time required for the preprocessing of reconstruction by the server using as input the number of captured images, the resolution of the captured images, and the available computational resources of the server, or a regression model that uses as input the number of captured images, the resolution of the captured images, and the available computational resources of the server to regression-predict the preprocessing time by the server; a prediction unit that, after the imaging by the moving object is performed, inputs, for each of the partial regions, the number of the acquired images in which the partial region is to be imaged, the resolution of the acquired images, and a default allocation of the server's computational resources into the learning model or the regression model, and predicts a result output by the learning model or the regression model as the preprocessing time; the determination unit determines the 3D reconstruction plan, including at least one of determining a processing priority of the server and allocating computational resources of the server, for each of the partial regions associated with each of the execution units, further based on the predicted preprocessing time, on the condition that the delivery date is met. The information processing device according to claim 12.

14. The processor: receiving basic requirements including an imaging target area, which is an area to be imaged by a moving body, a schedule including a period during which the moving body performs imaging and an interval during which the moving body repeats imaging within that period, a delivery time for providing a three-dimensional structure reconstructed from the captured images captured by the moving body, and service level requirements including quality related to the positional accuracy of the three-dimensional structure after reconstruction; generating an acquisition plan including a route of the mobile object, with the basic requirements being satisfied; For each of the imaging execution units defined in the schedule for the plurality of basic requirements, on the condition that each delivery date included in the service level requirements corresponding to each basic requirement is met, a step of determining a three-dimensional reconstruction plan for each of the execution units, the acquisition plan including the imaging start time of the moving body, the transmission time period for the acquired images captured by the moving body, and at least one of determining the processing priority of the server and allocating the computational resources of the server, based on (1) each of the acquisition plans, (2) movement constraints including at least one of the time periods and areas in which the moving body can move set up in each of the imaging target areas, and (3) at least one of the network resources of a hangar that stores the moving body and the computational resources of a server that reconstructs the three-dimensional structure; An information processing method that performs the above.

15. On the computer, a function for receiving basic requirements including an imaging target area, which is an area imaged by a moving body, a schedule including a period during which the moving body performs imaging and an interval during which the moving body repeats imaging within that period, a delivery time for providing a three-dimensional structure reconstructed from the captured images taken by the moving body, and service level requirements including quality related to the positional accuracy of the three-dimensional structure after reconstruction; a function of generating an acquisition plan including a route of the moving object, on the condition that the quality is satisfied with respect to the basic requirements; a function for determining a three-dimensional reconstruction plan for each of the execution units of imaging defined in the schedule of a plurality of the basic requirements, based on (1) each of the acquisition plans, (2) movement constraints including at least one of time periods and areas in which moving bodies set in each of the imaging target areas can move, and (3) at least one of network resources of a hangar that stores the moving bodies and computational resources of a server that reconstructs the three-dimensional structure, on the condition that each delivery date included in the service level requirements corresponding to each basic requirement is met, the acquisition plan, and at least one of the start time of imaging of the moving body, the time period for transmitting the acquired images captured by the moving body, and determination of the processing priority of the server and allocation of computational resources of the server; A program to make this happen.

Images