Control system, control method, and control program

Abstract

[Problem] To provide a system capable of more efficiently using acquired data acquired by an aircraft. [Solution] The present invention is a control system that is provided with: a data acquisition unit that acquires data to be determined, such as sensing data of a target region, acquired using a sensor mounted on a flying body; a data suitability determination unit that determines the suitability of the data to be determined that is acquired by the data acquisition unit; and an accompanying information generation unit that, when it is determined that the data is unsuitable by the data suitability determination unit, generates accompanying information including information relating to the content of the suitability determination. The control system associates the unsuitable data, which has been determined to be unsuitable, or the ground positions in the target region related to the suitability determination, with the accompanying information, and records the associated information in the recording unit or displays the associated information on the display unit.

Description

Control system, control method, and control program

[0001] The present invention relates to a control system, a control method, and a control program.

[0002] In recent years, there has been a growing interest in the practical application of aerial sensing systems that use aircraft to acquire images and spatial data of target areas on the ground from the sky. Aerial sensing requires that sensing be performed according to pre-set flight and measurement conditions so that the aerial sensing data can be used for the desired purpose. However, due to factors such as weather conditions and equipment status, it may be temporarily impossible to acquire aerial sensing data that meets the above-mentioned conditions during aerial sensing. Aerial sensing data generally comes in huge volumes, and the task of identifying aerial sensing data that does not meet the above-mentioned conditions from among the vast amount of aerial sensing data has taken a significant amount of time. Therefore, there has been a demand for a function that can identify aerial sensing data that does not meet the specified conditions.

[0003] Patent document 1 discloses a technology in which, when an unmanned aircraft conducting aerial photography moves a predetermined distance away from the planned flight route, a flag is embedded in the data of the aerial footage, thereby informing the user that the aerial footage at the time the flag is embedded needs to be edited.

[0004] Re-table No. 2018-198317

[0005] Patent Literature 1 discloses determining whether or not a photographing failure has occurred based on the actual flight position of the aircraft, and incorporating information about the photographing failure into the aerial video data, but only an either-or decision is made: if the photographing failure occurs, the photographing data is not used, and if the photographing failure occurs, the photographing data is used.As a result, even if the data could be used without problems depending on the data usage purpose and data processing content, data determined to be a photographing failure is not used, making it impossible to efficiently use aerial sensing data and quickly provide aerial sensing data or processed data to users.

[0006] Therefore, an object of the present invention is to provide a system that can more efficiently utilize data acquired by an aircraft.

[0007] According to the present invention, a control system is obtained which comprises a data acquisition unit that acquires data to be judged, which is at least one of sensing data including images or point cloud information of a target area acquired using a sensor mounted on an aircraft, data in process generated in the process of data processing of the sensing data, and processed data generated by data processing of the sensing data, a data suitability judgment unit that judges the suitability of the data to be judged acquired by the data acquisition unit, and an accompanying information generation unit that generates accompanying information including information regarding the content of the suitability judgment when the data suitability judgment unit judges the data to be inappropriate, and which associates the inappropriate data judged to be inappropriate, or the ground position of the target area related to the inappropriate judgment, or the position of the aircraft, or the control target position of the aircraft, or the control target route of the aircraft, with the accompanying information and records it in a recording unit or displays it on a display unit.

[0008] According to the present invention, data acquired by aircraft can be used more efficiently, thereby quickly providing data to users.

[0009] 1 is an overall configuration diagram of an aviation data processing system according to one embodiment of the present invention. FIG. 2 is a functional configuration diagram of an aircraft according to one embodiment of the present invention. FIG. 3 is a functional configuration diagram of a data acquisition base system according to one embodiment of the present invention. FIG. 4 is a functional configuration diagram of a data processing base system according to one embodiment of the present invention. FIG. 5 is a configuration diagram of a distribution system according to one embodiment of the present invention. FIG. 6 is a functional configuration diagram of a data management system according to one embodiment of the present invention. FIG. 7 is a functional configuration diagram of a data acquisition management unit according to one embodiment of the present invention. FIG. 8 is a table showing an example of criteria for determining whether or not sensing data is appropriate according to one embodiment of the present invention. FIG. 9 is a diagram showing an example of criteria for determining a response action according to one embodiment of the present invention. FIG. 10 is a table showing an example of information acquired by a status information acquisition unit according to one embodiment of the present invention. FIG. 11 is a table showing an example of associated information generated by an associated information assignment unit according to one embodiment of the present invention. FIG. 12 is a diagram showing an example of a display when an appropriateness determination result display unit of the present invention displays a result of an appropriateness determination. FIG. 13 is a diagram showing an example of a display when a response action determination result display unit of the present invention displays a determination result of a response action. FIG. 14 is a diagram showing an example of a display when an associated information display unit of the present invention displays associated information in association with inappropriate data. FIG. 15 is a diagram showing an example of a display when an associated information display unit of the present invention displays associated information in association with a sensing target area on the ground. FIG. 16 is a diagram showing an example of a display when an associated information display unit of the present invention displays associated information in association with the position of an aircraft. FIG. 17 is a table showing an example of data processing content instructed to a data processing unit by a data processing execution command unit of the present invention. FIG. 1 is a diagram showing an example of a display when the distribution information generation unit of the present invention displays accompanying information in association with a ground sensing target area. FIG. 2 is a diagram showing an example of a display when the distribution information generation unit of the present invention displays accompanying information in association with the position of an aircraft. FIG. 3 is a diagram showing an example of the hardware configuration of the data management system and data processing management unit of the present invention. FIG. 4 is a flowchart showing the processing flow by the aviation data processing system of the present invention. FIG. 5 is a flowchart showing the processing flow performed by the data suitability determination unit in the data management system of the present invention when determining data suitability. FIG. 6 is a flowchart showing the processing flow performed by the response action determination unit of the present invention when determining response actions. FIG. 7 is a flowchart showing the processing flow performed by the data suitability determination unit in the data processing management unit of the present invention when determining data suitability.1 is a flowchart showing a processing flow for data processing and data suitability determination when the data processing center system of the present invention executes SfM processing. FIG. 2 is a diagram showing an example of doming prevention processing executed by the data processing unit of the present invention.

[0010] The details of the embodiments of the present invention will be described below. The present invention has the following configuration. [Item 1] A control system comprising: a data acquisition unit that acquires data to be judged, which includes sensing data including images or point cloud information of a target area acquired using a sensor mounted on an air vehicle, in-process data generated in the process of data processing the sensing data, and processed data generated by data processing of the sensing data; a data suitability determination unit that performs a suitability determination on the target data acquired by the data acquisition unit; and an associated information generation unit that generates associated information including information on the content of the suitability determination when the data suitability determination unit determines the data to be judged as inappropriate, and the inappropriate data determined to be inappropriate, or the ground position of the target area, the position of the air vehicle, the control target position of the air vehicle, or the control target path of the air vehicle related to the inappropriateness determination, is associated with the associated information and recorded in a recording unit or displayed on a display unit. [Item 2] The control system described in Item 1, wherein the associated information includes information on the reason why the data was determined to be inappropriate by the suitability determination. [Item 3] The control system according to item 1 or 2, wherein the accompanying information is displayed on at least one of a map screen, a three-dimensional model screen, and a three-dimensional space screen in association with the inappropriate data, or the ground position of the target position related to the inappropriateness judgment, or the position of the aircraft, or the control target position of the aircraft, or the control target path of the aircraft. [Item 4] The control system according to any of items 1 to 3, wherein the accompanying information includes information on the degree to which the inappropriate data exceeds a reference value for the suitability judgment when the inappropriate data is judged to be inappropriate by the suitability judgment. [Item 5] The control system according to any of items 1 to 4, wherein the accompanying information includes at least any of the sensing implementation date and time related to the acquisition of the sensing data, the aircraft identification information, the sensor identification information, the aircraft control information, the sensor control information, and status information of the sensing data.[Item 6] The control system according to any one of items 1 to 5, wherein the data suitability determination unit determines the suitability of the data to be determined, or displays candidates for inappropriate data, using preset determination conditions or artificial intelligence. [Item 7] The control system according to any one of items 1 to 6, wherein the data suitability determination unit determines the suitability of the data to be determined, based on suitability determination input information for the data to be determined, received from a user. [Item 8] The control system according to any one of items 1 to 7, wherein, when the data suitability determination unit determines the suitability of the sensing data, the determination conditions include at least one of a state of the sensing data, a state of the air vehicle, and a state of the sensor, or, when the data suitability determination unit determines the suitability of the processed data, the determination conditions include a state of the processed data. [Item 9] The control system according to any one of items 1 to 8, wherein, when the data suitability determination unit determines the suitability of the sensing data and the determination conditions include a state of the flying object, the determination conditions are set by comparing a state quantity related to the flying object with a reference value, or a past actual value, or a control target value for a state quantity of a control object related to the flying object. [Item 10] The control system according to any one of items 1 to 9, wherein, when the data suitability determination unit determines the suitability of the sensing data and the determination conditions include a state of the sensing data, the determination conditions are set by comparing a state quantity related to the sensing data with a reference value, or a state quantity of past sensing data, or a target value, or an acquisition rate of sensing data for a target area of ​​a sensing object. [Item 11] In the control system according to any one of Items 1 to 10, when the data suitability determination unit determines the suitability of the in-processing data or the processed data, the determination criterion is set by comparing a state quantity related to the in-processing data or the processed data with a reference value, or a state quantity of past processed data, or a target value.[Item 12] The control system according to any one of items 1 to 11, further comprising a response action determination unit that determines a response action to be taken when the data suitability determination unit determines that the data to be determined is inappropriate, and the control system includes information about the response action in the associated information or executes the response action. [Item 13] The control system according to any one of items 1 to 12, wherein the response action determination unit determines the response action according to at least one of the content of the data to be determined, the state of the flying vehicle, the state of the sensor, and the result of the suitability determination. [Item 14] The control system according to any one of items 1 to 13, wherein the response action determination unit determines the response action according to designated input information for the response action received from a user. [Item 15] In the control system described in any one of items 1 to 14, the response action includes at least one of not using the inappropriate data, substituting other data for the inappropriate data, notifying of restrictions on use of the processed data, notifying that the inappropriate data is included, specifying a data processing method, re-performing sensing by an airborne vehicle, specifying a data processing method according to a user-specified use, and specifying a display method for the processed data according to a user-specified use. [Item 16] In the control system described in any one of items 1 to 15, the associated information includes information related to the response action. [Item 17] In the control system described in any one of items 1 to 16, when the information on the response action included in the associated information is not using the inappropriate data, the associated information includes information on a target area related to the inappropriate data. [Item 18] In the control system according to any one of Items 1 to 17, when the information on the response action included in the accompanying information is a substitution of other data in place of the inappropriate data, information on the other data to be substituted is included in the accompanying information.[Item 19] A control system according to any one of items 1 to 18, wherein, when the information on the response action included in the associated information is a notification of restrictions on the use of the processed data, information on the restrictions on the use is included in the associated information. [Item 20] A control system according to any one of items 1 to 19, wherein, when the information on the response action included in the associated information is a designation of the data processing method, information on the designated data processing method is included in the associated information. [Item 21] A control system according to any one of items 1 to 20, wherein, when the information on the response action included in the associated information is a re-execution of sensing by the air vehicle, the associated information includes sensing re-execution schedule information including at least one of a planned position, a planned area, and a planned time for the sensing re-execution. [Item 22] The control system according to any one of items 1 to 21, wherein the control system receives from a user a usage determination condition for determining that the inappropriate data is usable by at least one of the response actions of a notification of usage restrictions on the processed data, a notification that the processed data contains inappropriate data, and a designation of a data processing method, and, when the inappropriate data is determined to be inappropriate by the suitability determination, determines the inappropriate data as usable in accordance with the usage determination condition, or displays the inappropriate data as usable. [Item 23] The control system according to any one of items 1 to 22, wherein the control system receives from a user a usage determination condition for determining that the inappropriate data is usable by at least one of the response actions of a notification of usage restrictions on the processed data, a notification that the processed data contains inappropriate data, and a designation of a data processing method, and, when the inappropriate data is determined to be inappropriate by the suitability determination, determines a response action for using the inappropriate data in accordance with the usage determination condition, or displays candidate response actions for using the inappropriate data.[Item 24] The control system according to any one of items 1 to 23, wherein when the suitability judgment results in an inappropriateness judgment, the control system judges or displays a response action according to a judgment condition for the response action received from a user or by using artificial intelligence. [Item 25] A control method, wherein a computer executes the following steps: a data acquisition step of acquiring judgment target data at least any of sensing data including images or point cloud information of a target area acquired by a sensor mounted on an air vehicle, in-process data generated in the process of data processing of the sensing data, and post-processing data generated by data processing of the sensing data; a data suitability judgment step of making a suitability judgment for the judgment target data acquired by the data acquisition step; an associated information generation step of generating associated information including information on the suitability judgment for inappropriate data judged to be inappropriate; and a step of associating the inappropriate data, or a ground position of the target area corresponding to the inappropriate data, or the position of the air vehicle, or a control target position of the air vehicle, or a control target route of the air vehicle, with the associated information, and recording the inappropriate data in a recording unit or displaying it on a display unit. [Item 26] A control program that causes a computer to execute the following steps: a data acquisition step of acquiring data to be judged, which is at least any of sensing data including images or point cloud information of a target area acquired using a sensor mounted on an aircraft, in-process data generated in the process of data processing the sensing data, and processed data generated by data processing of the sensing data; a data suitability determination step of determining whether the target data acquired by the data acquisition unit is suitable; an associated information generation step of generating associated information containing information regarding the suitability determination for inappropriate data that has been judged to be inappropriate; and a step of associating the inappropriate data, or the surface position of the target area corresponding to the inappropriate data, or the position of the aircraft, or the control target position of the aircraft, or the control target route of the aircraft, with the associated information, and recording the associated information in a recording unit or displaying it on a display unit.

[0011] <A. First Embodiment> Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In this specification and the drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted. Furthermore, the embodiment described below is merely an example, and other known elements or alternative means may be adopted depending on the application, purpose, scale, etc.

[0012] [A-1. Configuration] (A-1-1. Overview) Fig. 1 is a diagram showing the overall configuration of an aviation data processing system 1 (hereinafter also referred to as "this system 1") according to one embodiment of the present invention. As shown in Fig. 1, this system includes an aircraft 1000, a data acquisition base system 2000, a data processing base system 3000, a distribution system 4000, and a satellite system 6000.

[0013] The aircraft 1000 acquires acquired data by sensing information about a target area from the sky using sensors including optical cameras, infrared cameras, and laser sensors such as LiDAR. If the aircraft is an unmanned aircraft, the data acquisition base system 2000 is a base equipped with a remote control device that exchanges control information with the aircraft, and is configured as a mobile vehicle-type or a stationary, immobile base. The data processing base system 3000 acquires the acquired data acquired by the aircraft, processes the data, and generates processed data. The distribution system 4000 acquires the processed data and distributes it to a user terminal device 5000 via an Internet connection.

[0014] The crisis management headquarters terminal, which is an example of a user terminal device, is a terminal device used by users such as the person in charge of the crisis management headquarters, and processed data can be viewed by accessing the distribution system from the crisis management headquarters terminal. In addition, the data acquisition base system 2000 and the data processing base system 3000 are provided with display devices and user input devices for users who manage or operate these systems.

[0015] (A-1-2. Aircraft 1000) Figure 2 is a functional configuration diagram of an aircraft according to one embodiment of the present invention. The aircraft 1000 is a system that constitutes an aircraft, and includes a flight unit 1100 with flight capabilities, a sensing unit 1200 that performs sensing using sensors, a communication unit 1300 that communicates with a data acquisition base system via a satellite system 6000 within the aviation data processing system 1 or a mobile phone communication network, a state determination unit 1400 that determines each state of flight, sensing, and positioning, a sensing data recording unit 1500 that records acquired data, etc., and a state determination result recording unit 1630 that records information related to flight control and measurement control, as well as state determination results.

[0016] In this specification, the term "aircraft" refers to any flying object capable of autonomous attitude control, regardless of the power source (electric power, prime mover, etc.), the control method (wireless or wired, fully autonomous flight type or partially manual flight type, etc.), and whether manned or unmanned. Aircraft may also be referred to as unmanned aerial vehicles (UAVs), flying objects, multicopters, RPAS (remote piloted aircraft systems), or UAS (unmanned aircraft systems). Aircraft may be fixed-wing type with fixed wings, multicopter type with multiple propellers, or VTOL type with both fixed wings and multiple propellers.

[0017] (A-1-2-1. Flight Unit 1100) The flight unit 1100 includes a self-position / velocity determination unit 1110, an attitude determination unit 1120, and a flight control unit 1130.

[0018] The aircraft's own position / speed determination unit 1110 has a function of determining the position and speed of the aircraft 1000. For example, the aircraft's position (absolute position) is measured based on satellite signals received from artificial satellites by a satellite signal antenna mounted on the aircraft using a GPS (Global Positioning System) or a GNSS (Global Navigation Satellite System). The aircraft's own position and speed can also be determined using, for example, an RTK-GNSS (Real Time Kinematic - Global Navigation Satellite System). The position information includes at least two-dimensional coordinate information (e.g., latitude and longitude) in a planar view, and preferably includes three-dimensional coordinate information including altitude information.

[0019] Next, the attitude determination unit 1120 has a function of determining the attitude (orientation) of the aircraft 1000. Specifically, the attitude of the aircraft is determined by a GPS compass consisting of a pair of satellite antennas mounted on the aircraft, a geomagnetic sensor, or the like.

[0020] Flight control section 1130, in turn, includes a processing unit, also referred to as a flight controller. The processing unit may include one or more processors, such as a programmable processor (e.g., a central processing unit (CPU), MPU, or DSP). The processing unit has access to memory. The memory stores logic, code, and / or program instructions that the processing unit can execute to perform one or more steps.

[0021] The processing unit has a function of controlling the flight state of the aircraft by controlling the output of the thrust generating device. Specifically, the processing unit controls the thrust generated from the propeller using the propeller, the motor that drives the propeller, and the motor control unit that controls the motor output, thereby adjusting the spatial arrangement, attitude angle, angular velocity, angular acceleration, angular jerk rate, and / or acceleration of the aircraft, which has six degrees of freedom (translational motion x, y, and z, and rotational motion θx, θy, and θz). In other words, the flight control unit 1130 controls the aircraft to perform various operations such as liftoff, forward movement, turning, and landing, and controls the attitude angle control and flight operations of the aircraft from takeoff to flight and landing, thereby controlling the flight state of the aircraft.

[0022] (A-1-2-2. Sensing Unit 1200) The sensing unit 1200 includes a sensor 1210, a sensor attitude control unit 1220, and a sensor control unit 1230.

[0023] The sensor 1210 is composed of, for example, an optical camera that acquires optical images, an infrared camera that acquires infrared images, a laser sensor such as LiDAR that acquires point cloud data, etc. The sensor acquires optical images, infrared images, point cloud data, etc. of the ground surface of a sensing target area such as a disaster area from the sky above the target area.

[0024] The sensor attitude control unit 1220 controls the relative angle of the sensor with respect to the aircraft body, for example, by controlling at least one of the angles around the three axes of a gimbal that is provided between the sensor and the aircraft and supports the sensor, thereby controlling the relative angle of the sensor with respect to the aircraft body.

[0025] The sensor control unit 1230 has a function of changing the timing and measurement parameters of sensing performed by the sensor 1210. For example, if the sensor is an optical camera, it controls the image acquisition timing, exposure, shutter speed, aperture (F-number), zoom amount, ISO, resolution, etc.

[0026] (A-1-2-3. Communication Unit 1300) The communication unit 1300 includes a control communication unit 1310, a data communication unit 1320, and a status data communication unit 1330. The communication unit 1300 is capable of direct communication, radio wave communication via a satellite system, or radio wave communication via a communication network NW, and includes, for example, a radio wave communication module. The communication unit 1300 is capable of communication with a data acquisition site system, etc. The communication unit 1300 is equipped with a communication function for wirelessly communicating with the data acquisition site system, etc., using, for example, Wi-Fi, 2.4 GHz, or a frequency band of 5.6 to 5.8 GHz. The communication unit 1300 is also equipped with a wireless communication function for communicating with the data acquisition site system, etc., via a communication network NW using a communication standard such as LTE (Long Term Evolution).

[0027] The control communication unit 1310 performs communication related to aircraft flight control exchanged between the aircraft 1000 and the data acquisition base system 2000. For example, if the aircraft is an unmanned aircraft and is remotely controlled from the data acquisition base system, the control communication unit 1310 receives flight control commands transmitted from the data acquisition base system to the aircraft. The control communication unit 1310 also transmits information related to the aircraft's flight status, including the aircraft's position, speed, and attitude determined by the flight unit 1100, from the aircraft to the data acquisition base system.

[0028] The data communication unit 1320 transmits the acquired data acquired by the sensing unit 1200 to a data acquisition base system, etc. The status data communication unit 1330 transmits the determination results of the flight status, sensing status, and communication status determined by the status determination unit 1400 (described later) to the data acquisition base system.

[0029] (A-1-2-4. State Determination Unit 1400) The state determination unit 1400 includes a flight state determination unit 1410, a sensing state determination unit 1420, and a communication state determination unit 1430.

[0030] The flight state determination unit 1410 determines that a flight abnormality has occurred when the position or speed of the aircraft determined by the aircraft's own position / speed determination unit 1110, or the attitude of the aircraft determined by the attitude determination unit 1120, deviates from a preset normal range. In addition, even if the position, speed, or attitude does not deviate from the normal range, the flight state determination unit 1410 also determines that a flight abnormality has occurred when at least one of the position, speed, or attitude does not normally follow the control command value from the flight control unit 1130.

[0031] The sensing state determination unit 1420 determines that a sensing abnormality has occurred when no data has been acquired by the sensor 1210 of the sensing unit 1200 or when an abnormality is found in the acquired data. Alternatively, a sensing abnormality may be determined when the sensor orientation control unit 1220 does not operate correctly in response to a control command value.

[0032] The communication status determination unit 1430 determines the communication status between the aircraft and an external system (such as a data acquisition base system or a satellite system) via the communication unit 1300. The communication status to be determined is, specifically, the communication speed (e.g., bps), and may include communication strength (e.g., dB) in addition to the communication speed. The communication status determination unit 1430 particularly determines the data communication speed of data acquired by the data communication unit 1320. Furthermore, when the aircraft position / velocity determination unit 1110 uses GPS or GNSS, or when the attitude determination unit 1120 uses a GPS compass, the communication status determination unit 1430 may determine the reception strength, positioning accuracy, and measurement accuracy of satellite signals received from artificial satellites via GPS or GNSS. Here, the positioning accuracy can be determined based on the strength of the satellite signals, the number of satellites from which satellite signals can be received, and the elevation angle of the satellites.

[0033] (A-1-2-5. Sensing data recording unit 1500) The sensing data recording unit 1500 records the acquired data acquired by the sensing unit 1200. The recorded acquired data is transmitted to a data acquisition site system or the like by the data communication unit 1320. The sensing data recording unit 1500 is configured by a storage device such as a RAM or ROM.

[0034] (A-1-2-6. Status data recording unit 1600) The status data recording unit 1600 includes a flight control information recording unit 1610, a measurement control information recording unit 1620, and a status determination result recording unit 1630. The status data recording unit 1600 is configured by a storage device such as a RAM or ROM.

[0035] The flight control information recording unit 1610 records information on the aircraft's position and speed determined by the self-position / speed determination unit 1110, and the aircraft's attitude determined by the attitude determination unit 1120. In addition to the aircraft's position, speed, and attitude, other state quantities measured by the flight unit 1100, such as acceleration, angular velocity, and angular acceleration, may also be recorded.

[0036] The measurement control information recording unit 1620 records status information on the relative angle of the sensor with respect to the aircraft body, which is acquired by the sensor attitude control unit 1220. It also records information on the timing of sensing execution by the sensor 1210 controlled by the sensor control unit 1230 and measurement parameters (image acquisition timing, exposure, shutter speed, aperture (F-number), zoom amount, ISO, and resolution). It also acquires information on whether sensing is being executed by the sensor 1210 from the sensor control unit 1230, or acquires information on whether sensing data is being recorded from the sensing data recording unit 1500, and records this as sensing data presence / absence information.

[0037] The state determination result recording unit 1630 records the determination results determined by the flight state determination unit 1410 , the sensing state determination unit 1420 , and the communication state determination unit 1430 of the state determination unit 1400 .

[0038] (A-1-3. Data Acquisition Base System 2000) Figure 3 is a configuration diagram of a data acquisition base system according to one embodiment of the present invention. The data acquisition base system 2000 includes a communications infrastructure management system 2100, an aircraft flight operation system 2200, a data management system 2300, an flight management system 2500, and an airspace monitoring system 2600.

[0039] If the aircraft is an unmanned aircraft, the data acquisition base system 2000 is a system that exchanges control information regarding the aircraft's flight section and sensing section with the aircraft, enabling remote control or data acquisition by a pilot (user) using the data acquisition base system. On the other hand, if the aircraft is a manned aircraft, the system exchanges control information and aircraft status information regarding the aircraft's flight section and sensing section with the aircraft, supporting data acquisition by the aircraft. The data acquisition base system may be composed of a mobile vehicle, ship, aircraft, etc., or may be composed of a non-moving, immovable building (fixed type).

[0040] The communication infrastructure management system 2100 is a system that manages communication means for data (acquired data acquired by sensing operations of the aircraft, associated information generated by the data management system, etc.) and control communication information transmitted and received between the data acquisition base system 2000, the aircraft 1000, and the data processing base system 3000. The communication infrastructure management system 2100 transmits, for example, information generated by the aircraft flight operating system 2200 and the data management system 2300 (described later) to the aircraft 1000.

[0041] Furthermore, there are multiple communication means for communicating with the aircraft 1000, such as a communication means for directly communicating with the aircraft 1000 using antenna equipment within the data acquisition base system 2000, a communication means via a mobile phone communication network (such as LTE), and a communication means for communicating via satellite relay via the satellite system 6000. Therefore, the communication infrastructure management system 2100 monitors the communication speed and strength of these communication means and switches between communication means. The transmission process may be, for example, parallel transmission or switched transmission. Furthermore, if the communication infrastructure management system 2100 determines through the data management system 2300 (described later) that it is appropriate to transmit the data by physically transporting the memory by a person, it may display this information on an appropriate display unit and request the administrator to physically transport the memory.

[0042] The aircraft flight operating system 2200 is a system that generates a mission for the aircraft to be operated and controls the movement of the aircraft. The mission is, for example, a movement plan including the movement route and movement speed of the aircraft 1000, and the movement route is generated in an airspace at an altitude of, for example, 150 m or less to avoid interference with passenger aircraft and the like. The aircraft flight operating system 2200 transmits a control signal to the aircraft 1000 via the communication infrastructure management system 2100 to operate the aircraft automatically.

[0043] The data management system 2300 acquires acquired data and status data of the target area sensed by aircraft via communication infrastructure management, determines the suitability of the content of the acquired data and the status data at the time of sensing, and then generates accompanying information. The detailed functions of the data management system 2300 will be described later.

[0044] The flight operations management system 2500 is a system that makes decisions and gives instructions for the operation of the aircraft 1000. The flight operations management system 2500 prepares plans for the aircraft 1000, including, for example, sensing and flight, and transmits the plans to the aircraft 1000 via the communication infrastructure management system 2100. The flight operations management system 2500 may also determine the priorities of the tasks that require processing and transmit them to the aircraft 1000. The flight operations management system 2500 may prepare plans for multiple aircraft 1000 and transmit information about the plans to each aircraft 1000.

[0045] The airspace monitoring system 2600 is a system that monitors the airspace in which the aircraft 1000 to be piloted is flying. The airspace monitoring system 2600 acquires information from an existing air traffic control system. The air traffic control system may be, for example, a drone traffic management system (UTM), an air traffic management subsystem (UASSP), or an air traffic management system (ATM). That is, the airspace monitoring system 2600 measures or acquires information about the environment and other aircraft in the airspace in which the aircraft 1000 is flying, and transmits the information to the flight management system 2500. If there is a problem with the work plan for the aircraft 1000 based on the information from the airspace monitoring system 2600, the flight management system 2500 changes the work plan.

[0046] (A-1-4. Data Processing Base System 3000) Fig. 4 is a configuration diagram of a data processing base system according to one embodiment of the present invention. The data processing base system comprises a communication unit 3100, a data processing unit 3200, and a data processing management unit 3300.

[0047] The communication unit 3100 transmits and receives data (acquired data and associated information) between the data processing base system 3000 and external systems (the data acquisition base system 2000 and the distribution system 4000). The communication unit 3100 receives, for example, acquired data and associated information from the data acquisition base system 2000. The communication unit 3100 also transmits processed data generated in the data processing base system 3000 to the distribution system 4000.

[0048] The data processing unit 3200 executes data processing based on the acquired data acquired by the aircraft 1000 and the associated information generated by the data management system.

[0049] The data processing management unit 3300 determines the appropriateness of the processed data generated by the data processing unit 3200 or the content of the data being processed and the corresponding action to be taken, then generates associated information and provides the processed data and the associated information to the distribution system. Details of the data processing management unit 3300 will be described later.

[0050] (A-1-5. Distribution System 4000) Fig. 5 is a configuration diagram of a distribution system according to one embodiment of the present invention. The distribution system 4000 comprises a communication unit 4100 and a distribution data management unit 4200.

[0051] The communication unit 4100 receives processed data and associated information from the data processing base system 3000. The received processed data and associated information may be data that has undergone web optimization processing in the data processing unit 3200 of the data processing base system 3000. The communication unit 4100 also distributes the processed data to user terminal devices 5000, including crisis management headquarters terminals, via an Internet line.

[0052] The distribution data management unit 4200 records the processed data received via the communication unit 4100, and distributes the latest processed data recorded in the distribution data management unit 4200 to user terminal devices, including device countermeasure headquarters terminals, accessed via an Internet line.

[0053] 6 is a functional configuration diagram of a data management system according to one embodiment of the present invention. The data management system 2300 includes a user request acquisition unit 2310, a sensing data acquisition unit 2320, a status information acquisition unit 2330, a data appropriateness determination unit 2340, a corresponding action determination unit 2350, an associated information assignment unit 2360, and a user input reception unit 2370.

[0054] (A-1-6-1. User request acquisition unit 2310) The user request acquisition unit 2310 acquires request information from the user in advance. The request information acquired by the user request acquisition unit 2310 includes suitability determination criteria for the suitability determination of sensing data executed by the data suitability determination unit 2340, which will be described later. In this case, for example, the suitability determination criteria include determination criteria related to the state of sensing data from the aircraft, the state of the aircraft, the state of the sensors, etc. For example, the suitability determination criteria include each of the inappropriateness determination criteria described in the user request information shown in Table T101 in FIG. 8, which will be described later.

[0055] Here, when the sensing data is an image taken by a camera, the criteria for determining whether the sensor state is inappropriate can be obtained as user-requested information, such as the resolution r of the camera image being below a reference value P1, or the brightness Y of the camera image being below a reference value L1, or the camera image being out of focus, and other criteria resulting from the sensor's setting parameters.

[0056] Furthermore, as criteria for determining whether the state of the aircraft is inappropriate, criteria for determining the flight state of the aircraft can be acquired as user request information, such as the absolute value of the difference between the position X of the aircraft and the target position X1 being equal to or greater than a reference value, the absolute value of the difference between the flight speed V of the aircraft and the target speed V1 being equal to or greater than a reference value, the absolute value of the difference between the flight altitude h of the aircraft and the target altitude h1 being equal to or greater than a reference value, or the angular difference between the attitude angles (yaw, roll, pitch) of the aircraft and the target angle being equal to or greater than a reference value (for example, 15 degrees).

[0057] In addition, as criteria for determining whether the state of sensing data is inappropriate, for example, the sensing operation was not performed properly and there is missing data in the camera image, there are water droplets, condensation, or dirt on the lens when the camera image was acquired, or there are defects in the camera image due to clouds or fog, or the image of the target area cannot be acquired because the lens cover of the camera image is closed, etc., judgment criteria regarding the content of the sensing data can be obtained as user request information.

[0058] The request information acquired by the user request acquisition unit 2310 may include suitability determination criteria for determining suitability of the processed data executed by the processed data suitability determination unit 3330, which will be described later. In this case, for example, the suitability determination criteria include inappropriateness determination criteria regarding the state of the processed data.

[0059] The request information acquired by the user request acquisition unit 2310 includes criteria for determining a response action to be executed by the response action determination unit 2350, which will be described later. In this case, for example, the response action determination criteria include multiple criteria related to the state of sensing data from the aircraft, the state of the aircraft, the state of the sensors, etc., and response actions corresponding to each criterion. For example, the criteria include the respective criteria described in the response action determination criteria shown in Table T102 in Fig. 9, which will be described later.

[0060] Here, when the sensing data is an image captured by a camera, the corresponding action according to the sensor state caused by the sensor setting parameters, such as the resolution r of the camera image, the brightness Y of the camera image, and the focus error of the camera image, can be obtained as user request information as a criterion for determining the corresponding action according to the sensor state.

[0061] Furthermore, as criteria for determining a corresponding action according to the state of the aircraft, for example, the difference between the position X of the aircraft and the target position, the difference between the flight speed V of the aircraft and the target speed, the difference between the flight altitude h of the aircraft and the target altitude, the angle difference between the attitude angles (Yaw, Roll, Pitch) of the aircraft and the target angle, etc., can be acquired as user request information indicating a corresponding action according to the flight state of the aircraft.

[0062] In addition, as a criterion for determining response actions according to the state of the sensing data, response actions according to defects in the content of the sensing data can be obtained as user request information, such as missing data in the camera image due to the influence of clouds or fog, defects in the camera image due to water droplets, condensation, or dirt on the lens when the camera image was acquired, or the inability to acquire an image of the target area due to the camera lens cover being closed.

[0063] The above-mentioned response actions may include response actions that allow the use of inappropriate data determined to be inappropriate (e.g., "notifying restrictions on the use of processed data," "notifying that inappropriate data is included," "specifying a data processing method," "specifying a display method for processed data," etc.), and may also include response actions that prohibit the use of inappropriate data determined to be inappropriate (e.g., "not using inappropriate data," "substituting other data for inappropriate data," "re-performing sensing by aircraft"). In other words, the request information acquired by the user request acquisition unit 2310 may include criteria for determining whether to allow the use of inappropriate data determined to be inappropriate.

[0064] (A-1-6-2. Sensing data acquisition unit 2320) The sensing data acquisition unit 2320 acquires sensing data sensed by the sensing unit 1200 mounted on the aircraft via the communication infrastructure management system 2100. The sensing data acquired here is, for example, optical image data or point cloud data acquired by a laser sensor such as LiDAR.

[0065] (A-1-6-3. Status information acquisition unit 2330) The status information acquisition unit 2330 acquires the information recorded in the aircraft flight control information recording unit 1610, the measurement control information recording unit 1620, and the status judgment result recording unit 1630 via the communication infrastructure management system 2100.

[0066] FIG. 10 is a table illustrating an example of information acquired by a status information acquisition unit according to an embodiment of the present invention. Information acquired from the flight control information recording unit 1610 includes, for example, the aircraft's position, flight speed, flight altitude, and attitude angles (yaw, roll, and pitch) as shown in Table T103 of FIG. 10 , as well as state quantities such as acceleration, angular velocity, and angular acceleration. Information acquired from the measurement control information recording unit 1620 includes, for example, the sensor's image acquisition timing, exposure, shutter speed, aperture (F-number), zoom amount (which affects focus), ISO, resolution, and brightness as shown in Table T103 of FIG. 10 , as well as other measurement parameters (contrast, histogram, etc.), timing of sensing execution, and the relative angle of the sensor with respect to the aircraft's fuselage (yaw, roll, and pitch). Sensing data presence / absence information regarding the presence or absence of sensing data may also be acquired from the measurement control information recording unit 1620.

[0067] The information acquired from the state determination result recording unit 1630 is information on the determination results made by the flight state determination unit 1410, the sensing state determination unit 1420, and the communication state determination unit 1430 of the state determination unit 1400. In particular, the acquired determination results include the reception strength of satellite signals, positioning accuracy, and measurement accuracy.

[0068] (A-1-6-4. Data suitability determination unit 2340) The data suitability determination unit 2340 determines the suitability of the sensing data based on the content of the sensing data acquired by the sensing data acquisition unit 2320, or various status information (including the status of the aircraft and the status of the sensors) acquired by the status information acquisition unit 2330, or a combination of these, and displays the determination result. The determination may also be based on suitability determination criteria such as those shown in FIG. 8, or suitability determination may be performed using artificial intelligence. Alternatively, the determination may be based on input information of a determination result that the user has determined to be inappropriate. The data suitability determination unit 2340 includes a data analysis unit 2341, a status information analysis unit 2342, and a suitability determination result display unit 2343.

[0069] 8 is a table showing an example of criteria for determining whether or not sensing data is appropriate according to an embodiment of the present invention. Table T101 in FIG. 8 shows an example of criteria for determining whether or not sensing data is appropriate when the sensing data is an image captured by a camera. The criteria for determining whether or not sensing data is appropriate include criteria related to the sensor state, the aircraft state, and the sensing data state.

[0070] The appropriateness determination criteria can be set based on input information from the user acquired by the user request acquisition unit 2310, but are not limited to this and may be set in advance by the system. Alternatively, the user may select the criteria to be applied by inputting into a selection field on the left side of T101 from default setting criteria for each target state quantity that are set in advance by the system, or may set any conditions by inputting into a criteria field for inappropriateness determination on the right side of T101.

[0071] As an example of criteria for determining whether or not sensing data is appropriate, which differs from that shown in Figure 8, the user may input the intended use of the sensing data or the processed data (such as for inspecting features or for forest surveying), and the system may automatically generate criteria for determining whether or not the data is appropriate based on the intended use input.

[0072] First, the data analysis unit 2341 analyzes the content of the sensing data acquired by the sensing data acquisition unit 2320 and determines whether the sensing data is inappropriate in light of the criteria for determining inappropriateness regarding the state of the sensing data, such as those shown in Table T101 in Fig. 8. Table T101 lists criteria that can be used to determine inappropriateness regarding the state of the sensing data, based on an analysis of the content of the sensing data, such as missing data in part of the camera image due to the influence of clouds or fog, a defect in part of the camera image due to water droplets, condensation, or dirt on the lens when the camera image was acquired, or local insufficient brightness or blown-out highlights, failure to acquire an image of the target area due to the camera lens cover being closed, the brightness Y of the camera image being equal to or less than a reference value L1 or equal to or greater than a reference value L2, or the camera image being out of focus.

[0073] In addition to the above, the criteria for judgment may include the presence of distortion in the image, the image being blurred, the image being of an unnecessary location outside the sensing target area (such as the aircraft's turnaround route), the resolution, contrast, or histogram being below a predetermined value, information indicating no sensing data being received, or the relative angle of the sensor to the aircraft's fuselage (Yaw, Roll, Pitch) exceeding a reference value.

[0074] Furthermore, when sensing is performed using a laser sensor, the criteria for determining inappropriateness may include when the point cloud data does not contain specified feature points, when the laser reflectivity is below a specified value, when there is an area within the sensing target area where the point cloud has not been acquired, or when the point cloud is from an unnecessary location outside the sensing target area (such as an aircraft turnaround route).

[0075] When the data analysis unit 2341 makes an inappropriate judgment based on the state of the sensing data, in addition to the judgment method of comparing the state quantity related to the sensing data with a reference value such as that shown in T101, as described above, it is also possible to make the judgment by comparing the state quantity with that of past sensing data, or comparing the state quantity of the sensing data with a target value, or the acquisition rate of sensing data for the target area to be sensed, etc.

[0076] Next, the status information analysis unit 2342 analyzes the contents of the various status information acquired by the status information acquisition unit 2330, and determines whether the sensing data is inappropriate in light of the criteria for determining inappropriateness related to the sensor status or the aircraft status, as shown in Table T101 in Fig. 8. Table T101 shows, as the criteria for determining inappropriateness related to the sensor status, criteria resulting from the sensor setting parameters, such as the resolution r of the camera image being equal to or less than a reference value P1, the brightness Y of the camera image being equal to or less than a reference value L1 or being equal to or greater than a reference value L2, or the camera image being out of focus.

[0077] In addition to the above, the judgment criteria may include receiving information indicating no sensing data, the relative angle (yaw, roll, pitch) of the sensor relative to the aircraft body being outside the reference range, the absolute angle (yaw, roll, pitch) of the sensor being outside the reference range, and data not being obtained for the point that should be sensed.

[0078] Table T101 also shows criteria for determining whether an aircraft condition is inappropriate, such as the difference X between the aircraft position and the target position (or target route) being outside the reference value range, the difference V between the aircraft flight speed and the target speed being outside the reference value range, the difference h between the aircraft flight altitude and the target altitude being outside the reference value range, or the difference in angle between the aircraft's attitude angle (yaw, roll, pitch) and the target angle being outside the reference range (for example, 15 degrees or more).

[0079] In addition to the above, the criteria may include the reception strength of the satellite signal falling below a reference value, the measurement accuracy of the aircraft's position or speed falling below a reference value, the aircraft's state quantities such as acceleration, angular velocity, and angular acceleration falling outside a reference range, etc. In particular, when sensing is performed using a laser sensor, the criteria for determining inappropriateness may include the flight altitude being higher than a predetermined value, because reflected laser light cannot be obtained if the distance to the sensing object is too great.

[0080] When the status information analysis unit 2342 makes an inappropriateness judgment based on the status of the aircraft, in addition to the judgment method of comparing the status quantities of the aircraft with the reference values ​​as shown in T101 as described above, the inappropriateness judgment may be made by comparing with past actual values ​​for the aircraft or by comparing with the control target values ​​of the status quantities to be controlled for the aircraft.

[0081] The method for determining inappropriateness described above may be a method in which an image is determined to be inappropriate if it deviates from the judgment criteria even once, or a method in which an image is determined to be inappropriate if the number of times the judgment criteria have been deviated exceeds a predetermined number, if there is an area in part of the image that deviates from the judgment criteria and the area that deviates from the judgment criteria exceeds a predetermined area, or if the proportion of the area that occupies the entire image exceeds a predetermined proportion.

[0082] Next, the suitability determination result display unit 2343 displays the results of the above-mentioned inappropriateness determination. FIG. 12 is a diagram showing an example of a display when the suitability determination result display unit of the present invention displays the results of the suitability determination. In the example shown in FIG. 12, inappropriate data determined to be inappropriate is displayed in a identifiable manner compared to the sensing data that is the subject of the suitability determination. A user of the data management system 2300 can select any inappropriate data using the user input accepting unit 2370, which will be described later. When any inappropriate data is selected, the data content of the selected inappropriate data (image data if the inappropriate data is an image) is displayed, along with the reason for the inappropriateness determination (e.g., out of focus), the degree of exceedance of the reference value (e.g., large, medium, or small), the time the inappropriate data was sensed, and the coordinate information of the ground area corresponding to the inappropriate data.

[0083] In addition, when any inappropriate data is selected, the suitability judgment result display unit 2343 displays on the screen, together with the above information, an input button that allows the user to select whether or not to confirm the data as inappropriate, so that the user can operate it using the user input accepting unit 2370 described below.

[0084] (A-1-6-5. Response action determination unit 2350) The response action determination unit 2350 determines a response action or displays candidate response actions based on the contents of the sensing data acquired by the sensing data acquisition unit 2320, or the various status information (including the status of the aircraft and the status of the sensors) acquired by the status information acquisition unit 2330, or a combination of these, or the result of the inappropriateness determination by the data appropriateness determination unit 2340 described above. The response action determination unit 2350 includes a determination processing unit 2351 and a response action determination result display unit 2352. The determination processing unit 2351 determines the response action to be taken especially when the data appropriateness determination unit 2340 determines that the data is inappropriate.

[0085] 9 is a diagram showing an example of response action determination criteria according to an embodiment of the present invention. The determination processing unit 2351 may determine a response action based on the response action determination criteria such as those shown in Table T102 in FIG. 9, or may determine a response action using artificial intelligence or the like. Alternatively, the response action may be determined based on command input information for the response action received from the user.

[0086] Table T102 in Figure 9 shows an example of criteria for determining a response action when it is assumed that the sensing data is an image captured by a camera. The response action criteria include inappropriateness criteria related to the sensor status, the status of the aircraft, and criteria for determining a response action according to the content of the sensing data. However, the criteria for determining a response action are not limited to this, and criteria for determining a response action corresponding to the result of an inappropriateness determination may also be set.

[0087] This determination criterion can be set based on input information from the user acquired by the user request acquisition unit 2310, but is not limited to this and may be set in advance by the system. Alternatively, the user may select a criterion to be applied by inputting a value into the selection field on the left side of T102 from default setting criteria for each target state quantity that are set in advance by the system, or may set an arbitrary condition by inputting a value into the corresponding action determination field on the right side of T102.

[0088] As an example of criteria for determining a corresponding action for sensing data different from that shown in Figure 9, the user may input the intended use of the sensing data or the processed data (such as for feature inspection or forest surveying), and the system may automatically generate criteria for determining a corresponding action and a determination result according to the intended use input.

[0089] Response actions determined by the determination processing unit 2351 include, for example, not using inappropriate data determined to be inappropriate, substituting other data for the inappropriate data, notifying restrictions on the use of the processed data, notifying that the processed data contains inappropriate data, specifying a data processing method, re-performing sensing by aircraft, specifying a data processing method according to the use specified by the user, and specifying a display method for the processed data according to the use specified by the user.

[0090] The above-mentioned "specifying a data processing method according to the intended use specified by the user" and "specifying a display method for processed data according to the intended use specified by the user" mean, for example, specifying a data processing method or a display method for processed data according to the intended use input acquired by the user request acquisition unit 2310.

[0091] Here, the response actions to be determined include a response action that does not use the inappropriate data determined to be inappropriate, and a response action that allows the use of the inappropriate data. Response actions that do not use the inappropriate data include, for example, "not using the inappropriate data determined to be inappropriate," "substituting other data for the inappropriate data," and "re-performing sensing by aircraft." Response actions that allow the use of the inappropriate data include, for example, "notifying restrictions on the use of the processed data," "notifying that the processed data contains inappropriate data," "specifying a data processing method," "specifying a data processing method according to the use specified by the user," and "specifying a display method for the processed data according to the use specified by the user."

[0092] As a criterion for determining a response action, for example, for inappropriate data whose degree of exceedance from the inappropriateness judgment criterion is less than a predetermined value, the response action can be to allow the use of the inappropriate data, and for inappropriate data whose degree of exceedance from the inappropriateness judgment criterion is more than the predetermined value, the response action can be to not use the inappropriate data.

[0093] First, in Table T102, as a criterion for determining a response action regarding the state of sensing data, for example, when there is data loss in part of a camera image due to disturbances such as clouds or fog, the response action is determined to be "substitute other data" or "remeasurement." Here, "substitute other data" means substituting other data (such as geographical or map information or past sensing data) in place of inappropriate data. Also, "remeasurement" means re-performing sensing using an aircraft.

[0094] Furthermore, in Table T102, if there is a defect in part of the camera image due to water droplets, condensation, or dirt on the lens when the camera image was acquired, or due to local insufficient brightness or blown-out highlights, or if an image of the target area could not be acquired due to the camera lens cover being closed, the response actions that do not allow the use of inappropriate data are determined to be "substitution of other data" and "remeasurement."

[0095] Furthermore, if the brightness Y of the camera image is equal to or less than the reference value L1 or is equal to or greater than the reference value L2, the response action is determined to be "image adjustment by adjusting brightness." That is, the response action for allowing the use of inappropriate data determined to be inappropriate by the data appropriateness determination unit 2340 is determined to be "designation of data processing method." In other words, it is determined that the use of inappropriate data for data processing is allowed on the condition that image adjustment by adjusting brightness is performed as the data processing method to be specified.

[0096] Furthermore, if the camera image is out of focus due to factors such as the camera focus setting, the response actions for allowing the use of inappropriate data are "notification that inappropriate data is included" and "unavailable for use in person determination." Here, "notification that inappropriate data is included" means allowing the use of inappropriate data for data processing on the condition that the data processor or the user viewing the processed data is notified that inappropriate data is included. Furthermore, "unavailable for use in person determination" means allowing the use of inappropriate data for data processing on the condition that the data processor or the user viewing the processed data is notified that the processed data will not be used for purposes such as determining the presence of people from image data, as a usage restriction.

[0097] Next, in Table T102, as a criterion for determining a response action related to the state of an aircraft, for example, when the resolution r of the camera image is equal to or less than a reference value P1 and equal to or greater than a reference value P2, the response actions for allowing the use of inappropriate data are determined to be "notification that inappropriate data is included" and "unavailable for use in human detection." Also, when the resolution r of the camera image is less than the reference value P2, the response actions for not allowing the use of inappropriate data are determined to be "substitute other data" and "remeasure."

[0098] Next, in Table T101, as a criterion for determining a response action related to the aircraft status, for example, if the absolute value X of the difference between the aircraft position and the target position (or target route) is equal to or greater than a reference value X1 and equal to or less than a reference value X2, the response action for allowing the use of inappropriate data is determined to be "position correction processing of the ground surface photographed area according to the position error." In other words, it is determined that the use of inappropriate data for data processing is allowed on the condition that "position correction processing of the ground surface photographed area according to the position error" is performed as the specified data processing method. On the other hand, if the absolute value X of the difference between the aircraft position and the target position (or target route) is greater than X2, the response actions for not allowing the use of inappropriate data are determined to be "substitution of other data" and "remeasurement."

[0099] Furthermore, if the absolute value V of the difference between the aircraft's flight speed and the target flight speed is equal to or greater than the reference value V1 and equal to or less than the reference value V2, the response action to allow the use of inappropriate data is determined to be "notification that inappropriate data is included." On the other hand, if the absolute value V of the difference between the aircraft's flight speed and the target flight speed is greater than V2, the response actions to not allow the use of inappropriate data are determined to be "substitution of other data" and "remeasurement."

[0100] Furthermore, if the absolute value h of the difference between the aircraft's flight altitude and the target flight altitude is equal to or greater than a reference value h1 and equal to or less than a reference value h2, the response action to allow the use of inappropriate data is determined to be "notification that inappropriate data is included." On the other hand, if the absolute value h of the difference between the aircraft's flight altitude and the target flight altitude is greater than h2, the response actions to not allow the use of inappropriate data are determined to be "substitution of other data" and "remeasurement."

[0101] Furthermore, if the angular difference between the aircraft's attitude angles (yaw, roll, pitch) and the target angle is within a predetermined range (for example, 15 degrees or more and 30 degrees or less), the system determines that the corresponding action to allow the use of inappropriate data is "position correction processing of the ground surface photographed area according to the angle error." In other words, the system determines that the use of inappropriate data for data processing is allowed on the condition that "position correction processing of the ground surface photographed area according to the angle error" is performed as the specified data processing method. On the other hand, if the angular difference between the aircraft's attitude angles (yaw, roll, pitch) and the target angle is greater than 30 degrees, the system determines that the corresponding actions to not allow the use of inappropriate data are "substitution of other data" and "remeasurement."

[0102] As described above, in the example shown in Table T102 of Figure 9, the judgment processing unit 2351 determines the inappropriate data that can be used by determining either a response action that does not allow the use of inappropriate data or a response action that allows the use of inappropriate data.

[0103] Next, the response action determination result display unit 2352 displays the determination result of the response action determined by the determination processing unit 2351. Fig. 13 is a diagram showing an example of a display when the response action determination result display unit of the present invention displays the determination result of the response action. In the example shown in Fig. 13, "image processing for brightness adjustment" is displayed as the determination result of the response action. Other candidate response actions (notification to the user, not allowed for use by human judgment, not allowed to use the data, substitution of other data, etc.) are also displayed.

[0104] A user of the data management system 2300 can select any inappropriate data using the user input accepting unit 2370, which will be described later, and the determination result of the response action corresponding to the selected inappropriate data is displayed. Furthermore, when any inappropriate data is selected, the determination result of the response action described above and other candidates are displayed so that they can be selected, and a confirmation button for the response action is also displayed. The user can select any response action using the user input accepting unit 2370, and with a specific response action selected, can confirm the response action by operating the confirmation button for the response action.

[0105] (A-1-6-6. Accompanying information assigning unit 2360) The accompanying information assigning unit 2360 generates accompanying information including information related to the content of the suitability judgment made by the data suitability judgment unit 2340, and records the accompanying information in association with inappropriate data, etc., or displays it on a display unit. The accompanying information assigning unit 2360 includes an accompanying information generating unit 2361, an information assigning unit 2362, and an accompanying information display unit 2363, and generates accompanying information using the accompanying information generating unit 2361, and records and displays the accompanying information in association with inappropriate data, etc.

[0106] Fig. 11 is a table showing an example of associated information generated by the associated information adding unit according to one embodiment of the present invention. Table T104 in Fig. 11 shows an example of associated information generated by the associated information generating unit 2361, in which the associated information includes the result of the inappropriateness determination, information on the degree to which the inappropriate data exceeded the standard value for determining whether or not the data was appropriate, the reason for the inappropriateness determination, the measurement conditions for the inappropriate data, the details of the response action, and the corresponding area of ​​the inappropriate data.

[0107] The information on the "result of inappropriateness determination" included in the accompanying information is information that can identify the result of the determination of whether or not the sensing data has been determined to be inappropriate by the data appropriateness determination unit 2340. For example, it can be information indicating the presence or absence of an inappropriateness determination flag. Next, the "information on the degree to which the inappropriate data has exceeded the standard value for appropriateness determination" included in the accompanying information is information that indicates the extent to which the inappropriate data has exceeded the standard value for appropriateness determination when the data is determined to be inappropriate by the data appropriateness determination unit 2340. For example, it can be information on three levels of inappropriateness exceeding levels: high, medium, and low.

[0108] Next, the "reason for inappropriateness determination" included in the accompanying information is information regarding the reason for determining that the data is inappropriate when the data is determined to be inappropriate by the data appropriateness determination unit 2340, and is, for example, identification information of the determination criterion that corresponds to the inappropriateness determination criterion among the inappropriateness determination criteria such as those listed in Table T101 in Fig. 8. One example is "the difference between the aircraft position and the target position is equal to or greater than a reference value."

[0109] Next, the "measurement conditions of inappropriate data" included in the accompanying information is information related to the measurement conditions of the sensing data determined to be inappropriate by the data suitability determination unit 2340, and is information including, for example, the date and time of sensing, aircraft identification information, sensor identification information, aircraft control information, sensor control information, and sensing data status information. Specifically, this information is information acquired by the status information acquisition unit 2330.

[0110] Next, the "contents of the corresponding action" included in the accompanying information is information on the corresponding action determined by the corresponding action determination unit 2350, for example, information on the corresponding action shown in Table T102 of FIG.

[0111] Next, the "location / area related to the inappropriate data" included in the accompanying information is information on the location or area related to the inappropriateness judgment that was determined to be inappropriate. For example, this may be the sensing ground location sensed using the inappropriate data, or the position of the aircraft when sensing using the inappropriate data. If considering a case where the inappropriateness judgment is caused by the aircraft's position or attitude deviating from a reference range, it is desirable for the inappropriate data to be the ground location that should have been sensed, or the aircraft's control target position or control target route when sensing using the inappropriate data. Alternatively, if the data appropriateness judgment determines that the data is inappropriate due to a lack of sensing data (unsensing) at the ground location or aircraft position that should have been sensed, this may be the ground location that should have been sensed, or the aircraft's control target position or control target route.

[0112] Table T104 in FIG. 11 shows an example in which information on the three-dimensional coordinates of the four corners of the ground position sensed by the inappropriate data is displayed as the "position / area associated with the inappropriate data."

[0113] The following describes examples of combinations of response actions and other information included in accompanying information.

[0114] First, when the information on the response action included in the accompanying information is to not use the inappropriate data, the accompanying information includes information on the target area related to the inappropriate data. The information on the target area related to the inappropriate data is, for example, the sensing ground position sensed by the inappropriate data, or the position of the aircraft when sensing by the inappropriate data, the ground position that should have been sensed by the inappropriate data, or the control target position or control target route of the aircraft when sensing by the inappropriate data.

[0115] Next, when the information on the response action included in the accompanying information is a substitution of other data for the inappropriate data, information on the other data to be substituted is included in the accompanying information. This other data to be substituted may be, for example, geographic information (map information), image data acquired in the past, or spatial data including already created three-dimensional shape information.

[0116] Next, when the information on the response action included in the accompanying information is a notification of restrictions on the use of the processed data, information on the restrictions on the use is included in the accompanying information. Here, the information on the restrictions on the use is, for example, that the data cannot be used for image analysis purposes such as determining people. Inappropriate data determined to be inappropriate because the image is out of focus or the resolution is below a standard value is allowed to be provided to users as an orthoimage, but it is desirable to make it unusable for image analysis purposes such as determining people, since it is considered that there is a high possibility of false detection or non-detection of the detection target in image analysis due to the data not clearly capturing the situation in the target area.

[0117] Next, when the information on the response action included in the accompanying information is a designation of a data processing method, information on the designated data processing method is included in the accompanying information. Here, the information on the designated data processing method is, for example, image processing for adjusting brightness when the inappropriate determination is caused by a brightness abnormality, or position correction processing of the ground surface photographing area according to the position error when the inappropriate determination is caused by an aircraft position error, etc.

[0118] Next, when the information on the response action included in the accompanying information is the re-implementation of sensing by an aircraft, the accompanying information includes sensing re-implementation schedule information including at least one of the planned location, planned area, and planned time for the re-implementation of sensing. Here, the planned location for the re-implementation of sensing includes the ground location where sensing is planned or the target location of the aircraft at the time of sensing, the planned area includes the ground area where sensing is planned or the target flight area of ​​the aircraft at the time of sensing, and the planned time includes date and time information for the planned sensing, etc.

[0119] Next, a description will be given of a function of recording accompanying information in association with inappropriate data, etc., by the information attaching unit 2362. The information attaching unit 2362 associates inappropriate data determined to be inappropriate, or the surface position of a target area related to the inappropriate determination, or the position of an aircraft, or the control target position of an aircraft, or the control target route of the aircraft, with accompanying information and records the inappropriate data in a storage device.

[0120] Here, the ground position of the target area related to the inappropriateness determination includes, for example, the sensed ground position sensed using the inappropriate data, the position coordinates (latitude, longitude, height (LLH)) of the ground position that should have been sensed, or the position coordinates (latitude, longitude, height (LLH)) above the sensed position. Furthermore, the position of the aircraft related to the inappropriateness determination includes, for example, the position coordinates (latitude, longitude, height (LLH)) of the aircraft when the inappropriate data was acquired by sensing. Furthermore, the control target position of the aircraft related to the inappropriateness determination includes, for example, the control target position of the aircraft that should have been sensed when the inappropriate data was acquired by sensing. Furthermore, the control target route of the aircraft related to the inappropriateness determination includes the control target route of the aircraft that should have been sensed. Here, the control target route may be a flight mission expressed by waypoints including a target position and a target speed. Furthermore, the control target route and control target position are generally created in advance before the aircraft begins flight.

[0121] Next, the associated information display unit 2363 displays the associated information generated by the associated information generator 2361, thereby informing the user of the data management system 2300 of the content of the associated information. For example, the associated information is displayed on the screen of at least one of a map, a three-dimensional model, and a three-dimensional space display device in association with the inappropriate data or the ground coordinates (latitude, longitude, and altitude (LLH)) of the target location related to the inappropriate data or the inappropriateness determination, or the coordinates (latitude, longitude, and altitude (LLH)) of the aircraft's location, or the coordinates (latitude, longitude, and altitude (LLH)) of the aircraft's control target location, or the control target route of the aircraft. Here, the map refers to information in a two-dimensional or three-dimensional map format, and the three-dimensional model or three-dimensional space refers to point cloud data or three-dimensional information generated by SfM processing. Display examples of the associated information displayed by the associated information display unit 2363 are shown in FIGS. 14, 15, and 16. FIG. 14 is a diagram showing an example of a display when the associated information display unit of the present invention displays associated information with inappropriate data. 15 is a diagram showing a display example when the accompanying information display unit of the present invention displays accompanying information in association with a sensing target area on the ground. FIG. 16 is a diagram showing a display example when the accompanying information display unit of the present invention displays accompanying information in association with the position of an aircraft.

[0122] In the example shown in FIG. 14 , the accompanying information is displayed in association with inappropriate data (sensing data determined to be inappropriate). Here, the inappropriate data is image data, and the user can select any inappropriate data by operating the cursor. The above-mentioned accompanying information is displayed on the screen of the display device for the selected inappropriate data. In FIG. 14 , the accompanying information displayed includes the inappropriateness determination result, the reason for the inappropriateness determination, the corresponding action, the sensing time, the coordinates of the corresponding ground area, and substitute data.

[0123] In the example shown in Fig. 15, the accompanying information is displayed in map format on the screen of the display device in association with the sensing target area on the ground. In Fig. 15, the sensing range on the ground for each piece of sensing data is indicated by a dotted line, and the accompanying information is displayed in association with two sensing target areas on the ground corresponding to the inappropriate data. In this example, the accompanying information also displays the reason for the inappropriateness determination and the corresponding action.

[0124] In the example shown in Figure 16, the accompanying information is displayed in map format on the screen of the display device in association with the position of the aircraft. In Figure 16, the position of the aircraft when sensing of each sensing data was performed (e.g., coordinates of the actual position) is indicated by a circle, and the accompanying information is displayed in association with the positions of the aircraft at two locations corresponding to the inappropriate data. In addition, the accompanying information here displays the reason for the inappropriateness determination and the corresponding action. Note that the positions indicated by the circles in Figure 16 may be the control target positions on the control target route of the aircraft.

[0125] (A-1-6-7. User input accepting unit 2370) The user input accepting unit 2370 accepts user input regarding the suitability judgment result displayed by the suitability judgment result display unit 2343 and the response action judgment result displayed by the response action judgment result display unit 2352. As described above, the user input accepting unit 2370 can accept user input regarding the displayed suitability judgment result and confirm the inappropriate data. In addition, the user input accepting unit 2370 can accept user input regarding the displayed response action judgment result and confirm the response action.

[0126] The user input receiving unit 2370 is not limited to the above, but can accept user input that allows the user to select sensing data that the user has judged to be inappropriate regardless of the suitability judgment result and confirm the inappropriate data, and similarly can accept user input that allows the user to input a response action that the user has judged and confirm the response action.

[0127] (A-1-7. Data Processing Management Unit 3300) The data processing management unit 3300 determines the appropriateness of the processed data and the corresponding action based on the content of the processed data generated by the data processing unit 3200 or the data being processed, generates associated information, and provides the processed data and the associated information to the distribution system. FIG. 7 is a functional configuration diagram of a data acquisition management unit according to one embodiment of the present invention. As shown in FIG. 7, the data processing management unit 3300 includes an associated information display unit 3310, a data processing execution command unit 3320, a data appropriateness determination unit 3330 for the processed data, a corresponding action determination unit 3340, an associated information assignment unit 3350, a user input acceptance unit 3360, and a distribution information generation unit 3370.

[0128] (A-1-7-1. Accompanying information display unit 3310) The accompanying information display unit 3310, like the accompanying information display unit 2363 in the data management system 2300 described above, displays the accompanying information generated by the accompanying information generation unit 2361. However, the accompanying information display unit 3310 differs from the accompanying information display unit 2363 in that it displays the accompanying information to the user of the data processing management unit 3300 (that is, the user in charge of data processing), but the content of the displayed accompanying information may be the same.

[0129] Here, if the corresponding action included in the accompanying information is a specification of a data processing method, such as "image processing for adjusting brightness," "position correction processing of the ground surface photographed area according to position error," or "position correction processing of the ground surface photographed area according to angle error," the instruction for the data processing method is notified to the data processor by the accompanying information display unit 3310, and the data processor can control the data processing unit 3200 in accordance with the corresponding action.

[0130] The associated information display unit 3310 also displays associated information including the suitability judgment results and corresponding actions for the data in process and the processed data, which are generated in the associated information adding unit 3350 (described later). With this configuration, the data processing staff can take measures such as not using inappropriate data or performing resensing in accordance with the corresponding actions.

[0131] (A-1-7-2. Data processing execution command unit 3320) The data processing execution command unit 3320 determines the content of data processing to be executed by the data processing unit 3200, and sends an execution command for that data processing to the data processing unit 3200. FIG. 17 is a table showing an example of the data processing content that the data processing execution command unit of the present invention commands the data processing unit to. The data processing execution command unit 3320 can generate execution commands for various data processing such as those shown in Table T105 of FIG. 17. The content of the data processing for which an execution command is issued may be set in advance, or may be set by selecting a specific data processing from the list of data processing shown in Table T105.

[0132] The details of each data processing shown in Table T105 are as follows. ・Identification No. 001: A process of acquiring multiple optical images as acquired data, performing orthoimage conversion on the acquired optical images, and generating an orthoimage by integrating the multiple orthoimages. ・Identification No. 002: A process of acquiring multiple optical images as acquired data and generating three-dimensional spatial data from the multiple optical images using SfM (Structure from Motion) processing. ・Identification No. 003: A process of acquiring point cloud data as acquired data and generating three-dimensional spatial data from the point cloud data using DSM (Digital Surface Model) or DEM (Digital Elevation Model). ・Identification No. 004: A process of performing the orthoimage generation process shown in No. 001, acquiring geographic data as processing data, and generating integrated map data by integrating the orthoimage and geographic data. ・Identification No. 005: A process of performing the map integrated image generation process shown in No. 004, acquiring past integrated map images as processing data, and performing difference analysis between the current and past integrated map images. - Identification No. 006: Processing to generate the orthoimage shown in No. 001, obtain disaster information as data for processing, and generate a disaster information integrated image that integrates the orthoimage and disaster information. - Identification No. 007: Processing to generate the orthoimage shown in No. 001, obtain on-site unit information as data for processing, and generate a on-site unit information integrated image that integrates the orthoimage and on-site unit information.

[0133] (A-1-7-3. Data suitability determination unit 3330) The data suitability determination unit 3330 determines the suitability of data or displays the suitability determination result based on the content of the data being processed that is generated in the course of data processing by the data processing unit 3200 or the content of the processed data that is generated when data processing is completed. The data suitability determination unit 3330 includes a data analysis unit 3331 and a suitability determination result display unit 3332.

[0134] The data analysis unit 3331 analyzes the contents of the data in progress and the processed data acquired from the data processing unit 3200, and judges inappropriate data based on the judgment criteria related to the contents of the data in progress and the processed data. The judgment criteria for judging inappropriate data in the data analysis unit 3331 can be set for each of the above-mentioned data processing contents, for example, as follows:

[0135] The criteria for determining inappropriateness in data processing identification Nos. 001, 004, 005, and 006, which generate orthoimages, may include, for example, when lens distortion of a predetermined value or more is detected from the image data, when blurring or focus shift of the image data is detected, when a portion of the captured image is detected as missing, when an image in an unnecessary position (such as an aircraft's turnaround route) is detected, when insufficient brightness or blown-out highlights are detected in the image data, etc.

[0136] The criteria for determining whether a data processing identification No. 002 is inappropriate for SfM processing using multiple images can be, for example, when the object for which three-dimensional information is to be generated is not included in the image data, or when feature points of the object cannot be obtained. Other examples include detection of lens distortion exceeding a predetermined value, failure to capture ground control points (GCPs, check points, etc.) in the image data, detection of doming exceeding a predetermined value, detection of partial omissions in the captured image, detection of the presence of an image in an unnecessary position (such as an aircraft's return path), and detection of insufficient luminosity or blown-out highlights in the image data. Furthermore, the criteria for determining whether a data processing is inappropriate are not limited to those described above, and criteria can be established for each processing step of SfM processing.

[0137] The criteria for determining inappropriateness in data processing identification No. 003, which generates three-dimensional spatial data from a measurement point cloud, can be, for example, when an area is detected where the laser does not reach the ground surface due to topography such as a cliff, or where point cloud data cannot be acquired due to water, fog, or other objects that absorb the laser. It can also include when a part of the point cloud data is missing, or when feature points cannot be detected from the point cloud data.

[0138] In addition to the above, the criteria for determining inappropriateness in the data analysis unit 3331 may include the following: - When the data processing content cannot detect a pair of images contained in the image data that can be used as stereo images for measuring distances on the ground. - When processing both image data and point cloud data, if point cloud data is not obtained and the presence of clouds or fog is detected in the image.

[0139] The inappropriateness determination of the data in process and the processed data in the data analysis unit 3331 may be determined by comparing the state quantities of the data in process and the processed data with the reference value as described above, but may also be determined by comparing the state quantities with those of past processed data or with a target value.

[0140] Next, the appropriateness judgment result display unit 3332 displays the result of the above-mentioned inappropriateness judgment. Specifically, the display can be similar to the display of the inappropriateness judgment result displayed in the appropriateness judgment result display unit 2343 of the data management system 2300 shown in FIG. 12. In other words, inappropriate data judged to be inappropriate is displayed in an identifiable manner compared to the in-process data or processed data that is the subject of the appropriateness judgment. The user of the data processing management unit 3300 can select any inappropriate data displayed by the user input receiving unit 3360, which will be described later.

[0141] (A-1-7-4. Response action determination unit 3340) The response action determination unit 3340 determines a response action or displays candidate response actions based on the contents of the data being processed or the processed data, or a combination of these, acquired from the data processing unit 3200. The response action determination unit 3340 includes a determination processing unit 3341 and a response action determination result display unit 3342. The determination processing unit 3341 determines the response action to be taken, particularly when the data appropriateness determination unit 3330 determines that the data is inappropriate.

[0142] The response action determination unit 3340 determines a response action based on state quantities related to the data being processed and the processed data, and the determination result of the inappropriateness determination by the data appropriateness determination unit 3330. Similar to the determination processing unit 2351 of the data management system 2300, the response actions determined by the determination processing unit 3341 include, for example, not using the inappropriate data determined to be inappropriate, substituting other data for the inappropriate data, notifying the user of restrictions on the use of the processed data, notifying the user that inappropriate data is included, specifying a data processing method, re-performing sensing by an aircraft, specifying a data processing method according to the use specified by the user, and specifying a display method for the processed data according to the use specified by the user.

[0143] Here, when performing data processing identification No. 002, which performs SfM processing using multiple images, and doming of a predetermined value or more is detected due to an inappropriate judgment, the response action judgment unit 3340 judges that the response action is to take oblique images of the target area by "re-performing sensing by aircraft" and to perform doming response processing by "specifying a data processing method."

[0144] The judgment criteria of the corresponding action judgment unit 3340 can be set based on input information from the user acquired by the user request acquisition unit 2310, but are not limited to this and may be set in advance by the system. Alternatively, the user may select a criterion to be applied by inputting a value into the selection field on the left side of T102 from default setting criteria for each target state quantity that are set in advance by the system, or may set an arbitrary condition by inputting a value into the corresponding action judgment field on the right side of T102.

[0145] As an example of criteria for determining a corresponding action, the user may input the intended use of the processed data (such as for feature inspection or forest surveying), and the system may automatically generate criteria and results for determining a corresponding action based on the intended use entered.

[0146] Next, the response action determination result display unit 3342 displays the determination result of the response action determined by the determination processing unit 3341. The content displayed by the response action determination result display unit 3342 can be similar to the content displayed by the response action determination result display unit 2352 of the data management system 2300, as shown in FIG. 13 . A user of the data processing management unit 3300 can select any inappropriate data using the user input accepting unit 3360 (described later), and the determination result of the response action corresponding to the selected inappropriate data is displayed. Furthermore, when any inappropriate data is selected, the determination result of the response action described above and other candidates are displayed so that they can be selected, and a confirmation button for the response action is also displayed. The user can select any response action using the user input accepting unit 3360, and with a specific response action selected, can confirm the response action by operating the confirmation button for the response action.

[0147] (A-1-7-5. Accompanying information assigning unit 3350) The accompanying information assigning unit 3350 generates accompanying information including information related to the content of the suitability judgment made by the data suitability judgment unit 3330, and records the accompanying information in association with inappropriate data, etc., or displays it on a display unit. The accompanying information assigning unit 3350 includes an accompanying information generating unit 3351 and an information assigning unit 3352, and generates accompanying information using the accompanying information generating unit 3351, and records or displays the accompanying information in association with inappropriate data, etc.

[0148] The associated information generation unit 3351 generates associated information by adding or replacing the determination results by the data appropriateness determination unit 3330 and the corresponding action determination unit 3340 of the data processing management unit 3300 to the associated information generated in the data management system 2300 as shown in Fig. 11. Therefore, the associated information generated by the associated information generation unit 3351 can be associated information that includes at least one of or all of the appropriateness determination result and the corresponding action determination result generated in the data management system 2300 before the execution of data processing, and the appropriateness determination result and the corresponding action determination result generated during or after the execution of data processing.

[0149] The information attaching unit 3352 has the same function as the information attaching unit 2362 described above, and records the accompanying information generated by the accompanying information generating unit 3351 in association with inappropriate data, etc. The information attaching unit 3352 associates the inappropriate data determined to be inappropriate, or the surface position of the target area related to the inappropriateness determination, or the position of the aircraft, or the control target position of the aircraft, or the control target route of the aircraft, with the accompanying information and records it in a storage device.

[0150] The accompanying information generated by the accompanying information generation unit 3351 is displayed to the user of the data processing management unit 3300 (i.e., the data processing staff) by the aforementioned accompanying information display unit 3310 in association with the inappropriate data, etc. The accompanying information display unit 3310 associates the accompanying information with the inappropriate data determined to be inappropriate, or the surface position of the target area related to the inappropriate determination, or the position of the aircraft, or the control target position of the aircraft, or the control target route of the aircraft, and displays the accompanying information on a display device.

[0151] In this way, by displaying accompanying information including response actions for data being processed and processed data to the data processing staff of the data processing management unit 3300, the data processing staff can take action such as not using inappropriate data or conducting re-sensing in accordance with the response actions.

[0152] (A-1-7-6. User input accepting unit 3360) The user input accepting unit 3360 accepts user input regarding the suitability judgment result displayed by the suitability judgment result display unit 3332 and the response action judgment result displayed by the response action judgment result display unit 3342. As described above, the user input accepting unit 3360 can accept user input regarding the displayed suitability judgment result and confirm the inappropriate data. In addition, the user input accepting unit 3360 can accept user input regarding the displayed response action judgment result and confirm the response action.

[0153] The user input accepting unit 3360 is not limited to the above, but can accept user input that allows the user to select sensing data that the user has judged to be inappropriate regardless of the suitability judgment result and confirm the inappropriate data, and similarly can accept user input that allows the user to input a response action that the user has judged and confirm the response action.

[0154] (A-1-7-7. Distribution information generation unit 3370) The distribution information generation unit 3370 generates distribution information that associates the associated information generated by the associated information generation unit 3351 with the processed data generated by the data processing unit 3200, thereby making it possible to inform the user of the user terminal device that views the distribution information via the distribution system 4000 of the contents of the associated information corresponding to the processed data.

[0155] As an example, the distribution information generation unit 3370 generates distribution information in which accompanying information is displayed on the screen of the display device of the user terminal device in map format in association with inappropriate data determined to be inappropriate, or the ground position of the target position related to the inappropriate determination, or the position of the aircraft, or the control target position of the aircraft, or the control target route of the aircraft.

[0156] 18 and 19 show examples of display of distribution information generated by the distribution information generation unit 3370 on the display screen of a user terminal device. Fig. 18 is a diagram showing an example of display when the distribution information generation unit of the present invention displays accompanying information in association with a terrestrial sensing target area. Fig. 19 is a diagram showing an example of display when the distribution information generation unit of the present invention displays accompanying information in association with the position of an aircraft.

[0157] In the display example shown in Figure 18, the processed data is displayed in map format on the screen of a display device, and accompanying information is displayed in association with the sensing target area on the ground corresponding to the inappropriate data. Furthermore, in Figure 18, the ground range corresponding to each processed data is indicated by a dotted line, and accompanying information is displayed in association with the two ground ranges corresponding to the inappropriate data. The accompanying information also includes the reason for the inappropriateness determination and corresponding actions.

[0158] The display example shown in Figure 19 shows an example in which accompanying information is displayed on the screen of a display device where processed data is displayed in map format, in association with the position of the aircraft corresponding to the inappropriate data. Furthermore, in Figure 19, the position of the aircraft corresponding to each processed data (e.g., coordinates of the actual position) is indicated by a circle, and accompanying information is displayed in association with the positions of two aircraft corresponding to the inappropriate data. In addition, the accompanying information displayed here includes the reason for the inappropriateness determination, corresponding actions, etc. Note that the positions indicated by the circles in Figure 19 may be the control target positions on the control target route of the aircraft.

[0159] 18 and 19 show an example in which the reason for the inappropriateness determination is "out of focus," and the corresponding actions are "notify the user of the inappropriateness determination information" and "cannot be used for human detection." Therefore, a user who sees this distribution information can understand that the inappropriate data is processed based on sensing data that is out of focus. Furthermore, since the data cannot be used for human detection, the user can understand that the data cannot be used for determining the presence or absence of a person due to the influence of out of focus.

[0160] In this way, by displaying accompanying information in association with the processed data or information corresponding to the processed data, such as the ground location or aircraft location, to a viewing user viewing the processed data, the viewing user can correctly recognize the locations where data determined to be a photography failure is being used and the locations where it is not, and the reason why inappropriate data was determined to be inappropriate, and further, the range of applications in which the processed data can be used can be understood, so that the situation in the target area can be correctly understood based on the processed data.In addition, since it is possible to understand future response actions, such as whether the inappropriate data is scheduled to be updated by re-measurement, if the user who checked the processed data is a member of a crisis response headquarters, the user can more appropriately determine plans for future crisis response work.

[0161] <Hardware Configuration> Figure 20 is a diagram showing an example of the hardware configuration of a data management system 2300 and a data processing management unit 3300. Here, the data management system 2300 and the data processing management unit 3300 in the present invention are information processing devices such as a server device or a PC. As shown in the figure, the data management system 2300 and the data processing management unit 3300 have an input device 100, an output device 200, a processing device 300, a main memory device 400, an auxiliary memory device 500, a communication device 600, and a bus 700 that electrically connects these devices.

[0162] The input device 100 is a device that allows a user to input information and instructions to the data management system 2300 and the data processing management unit 3300. Specifically, the input device 100 is, for example, a touch panel, a keyboard, a mouse, or an audio input device such as a microphone.

[0163] The output device 200 is a device that outputs information generated by the data management system 2300 and the data processing management unit 3300. Specifically, the output device 200 is a display device (including eyewear, AR, VR display devices, etc.), a printer, or a speaker.

[0164] The processing device 300 is, for example, a device that performs arithmetic processing. Specifically, the processing device 300 is, for example, a CPU, a microprocessor, a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), or other semiconductor devices capable of performing calculations.

[0165] The main storage device 400 is a memory device such as a RAM that temporarily stores various read information and a ROM that stores programs, application programs, and other various information executed by the processing device 300. The auxiliary storage device 500 is a non-volatile storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), or a flash memory that can store digital information.

[0166] The communication device 600 is a device that performs wireless or wired information communication with an external device.

[0167] Although Figure 20 describes the hardware configuration of the data management system 2300 and the data processing management unit 3300, other systems within the data acquisition base system 2000 (communications infrastructure management system 2100, aircraft operation operating system 2200, operation management system 2500, airspace monitoring system 2600), other systems within the data processing base system 3000 (communications unit 3100, data processing unit 3200), distribution systems, and user terminal devices (including crisis management headquarters terminals) can also be realized with a hardware configuration similar to that shown in Figure 20 above.

[0168] <Control Flow> (A-1-8. Upper Control Flow of the Aviation Data Processing System) Next, the control flow of the aviation data processing system will be described with reference to the drawings. Figure 21 is a flowchart showing the processing flow of the aviation data processing system 1.

[0169] First, the user request information is acquired by the user request acquisition unit 2310 of the data management system 2300 (step 101). Note that this step may be performed before starting the sensing operation by the aircraft, or may be performed while the sensing operation is being performed.

[0170] Next, the sensing data acquisition unit 2320 and the status information acquisition unit 2330 of the data management system 2300 acquire sensing data and status information from the aircraft (step 102). As an example of the status information to be acquired, status information such as that shown in Table T103 in FIG. 10 is acquired.

[0171] Next, the data suitability determination unit 2340 of the data management system 2300 determines whether the data is inappropriate based on the acquired sensing data, status information, and other information (step 103). Here, after the aircraft has completed its flight for sensing operations, the data suitability determination in this step may be performed within the data acquisition base system 2000 based on data received via wireless communication or read from the sensing data recording unit 1500 and status data recording unit 1600 of the aircraft. Alternatively, it is also possible to mount each functional unit provided in the data acquisition base system 2000 on the aircraft and perform the data suitability determination in this step while the aircraft is in flight for sensing operations.

[0172] Next, as a result of the data suitability judgment by the data suitability judgment unit 2340, it is judged whether or not there is inappropriate data judged to be inappropriate, and the step to transition to is determined according to the judgment result (step 104). If there is inappropriate data, the process transitions to step 105, and if there is no inappropriate data, the process transitions to step 108.

[0173] Next, the response action determination unit 2350 of the data management system 2300 determines the response action to be taken when the data is determined to be inappropriate (step 105). This step may be performed within the data acquisition base system 2000 after the aircraft has completed its flight for sensing work, but it is also possible to implement the respective functional units provided in the data acquisition base system 2000 in the aircraft and perform the response action determination in this step while the aircraft is in flight for sensing work.

[0174] Next, the accompanying information adding unit 2360 of the data management system 2300 generates accompanying information including information on the suitability judgment and information on the content of the response action judgment, and records the accompanying information in association with the inappropriate data, etc. (step 106).

[0175] Next, the associated information assigning unit 2360 of the data management system 2300 displays the associated information in association with the inappropriate data, etc. to the user of the data management system 2300 (i.e., the person in charge of data sensing). Furthermore, the associated information display unit 3310 of the data processing management unit 3300 displays the associated information in association with the inappropriate data, etc. to the user of the data processing management unit 3300 (i.e., the person in charge of data processing) (step 107).

[0176] Next, the data processing execution command section 3320 of the data processing management section 3300 commands the data processing section 3200 to execute data processing (step 108).

[0177] Next, the data suitability determination unit 3330 of the data processing management unit 3300 determines whether the data being processed or the processed data is suitable (step 109).

[0178] Next, the data processing management unit 3300 determines whether there is any inappropriate data determined to be inappropriate as a result of the data appropriateness determination by the data appropriateness determination unit 3330, and determines the step to transition to based on the determination result (step 110). If there is inappropriate data, the process transitions to step 111, and if there is no inappropriate data, the process of this flowchart ends.

[0179] Next, the corresponding action determination unit 3340 of the data processing management unit 3300 determines the corresponding action to be taken when the data is determined to be inappropriate (step 111).

[0180] Next, the accompanying information adding unit 3350 of the data processing management unit 3300 generates accompanying information including information regarding the suitability judgment and information regarding the content of the response action judgment, and records the accompanying information in association with the inappropriate data, etc. (step 112).

[0181] Next, the accompanying information display unit 3310 of the data processing management unit 3300 displays the accompanying information in association with the inappropriate data, etc., to the user of the data processing management unit 3300 (that is, the person in charge of data processing) (step 113).

[0182] Next, the distribution information generation unit 3370 of the data processing management unit 3300 generates distribution information including the accompanying information and the processed data (step 114). Furthermore, the distribution information generated in this step is distributed to the user terminal device 5000 by the distribution system 4000.

[0183] (A-1-9. Processing Flow for Determining Data Appropriateness) FIG. 22 is a flowchart showing the processing flow performed when the data appropriateness determination unit 2340 in the data management system of the present invention determines the appropriateness of data.

[0184] First, the data analysis unit 2341 determines whether there is any missing data in the sensing data acquired from the sensing data acquisition unit 2320 (step 201). For example, if the sensing operation is not performed and no sensing data can be obtained, if the number of pieces of sensing data is clearly small, or if the lens cover remains closed and no ground image can be captured as shown in Fig. 8, it is possible to determine that the data is inappropriate due to missing data.

[0185] Next, the data analysis unit 2341 judges whether the sensing data is appropriate or not based on the content of the sensing data (step 202). For example, as shown in Fig. 8, if there are water droplets, condensation, or dirt on the lens when the camera image is acquired, or if there is a defective area in the camera image where the ground is not captured due to clouds or fog, etc., it is possible to judge whether the sensing data is appropriate or not based on the sensing data.

[0186] Next, the status information analysis unit 2342 determines suitability based on the status of the aircraft (step 203). For example, as shown in Fig. 8, suitability can be determined based on status quantities such as the aircraft's position, flight speed, flight altitude, and attitude.

[0187] Next, the state information analysis unit 2342 determines whether the sensor is suitable or not based on the state of the sensor (step 204). For example, as shown in Fig. 8, the determination can be made based on the sensor settings such as the resolution, focus, and brightness of the camera.

[0188] Next, it is determined whether or not there was an inappropriate judgment based on the results of the judgments made in steps 201 to 204, and the process to be transitioned to is determined based on the judgment result (step 205). If there is an inappropriate judgment, the process transitions to step 206, and if there is no inappropriate judgment, the process transitions to step 208.

[0189] Next, the inappropriate data that has been judged to be inappropriate and the details of the inappropriate judgment are displayed by the appropriateness judgment result display unit 2343 (step 206). For example, the inappropriate data as shown in Fig. 12, the reason for the inappropriateness judgment for the inappropriate data, the degree of exceedance of the reference value, the sensing time, the location of the target ground area, etc. are displayed.

[0190] Next, the user input receiving unit 2370 receives a confirmation input from the user for the displayed inappropriateness determination result (step 207). That is, by operating the confirmation button for inappropriate data as shown in Fig. 12, the inappropriateness determination result is confirmed for the sensing data determined to be inappropriate data by the inappropriateness determination and the content of the determination.

[0191] Next, the display device displays the sensing data, including the sensing data that has not been determined to be inappropriate in the data appropriateness judgment, to the user, regardless of the result of the appropriateness judgment (step 208).

[0192] Next, the user input receiving unit 2370 receives a selection input of sensing data to be determined as inappropriate from the user (step 209). In steps 208 and 209, the user can manually determine as inappropriate sensing data even for sensing data that has not been determined as inappropriate by the data appropriateness determining unit 2340, and input the determined data into the system.

[0193] Next, the data appropriateness judgment unit 2340 determines the inappropriate data and the content of the inappropriateness judgment based on the input information obtained from the user in steps 207 and 209, and records the determined inappropriate data and the content of the inappropriateness judgment (step 210).

[0194] (A-1-10. Processing Flow of Determining a Response Action) FIG. 23 is a flowchart showing the processing flow performed by the response action determining unit 2350 of the present invention when determining a response action.

[0195] First, the determination processing unit 2351 determines an action to be taken based on the data content of the sensing data acquired from the sensing data acquisition unit 2320 (step 301). For example, as shown in Fig. 9, it is possible to determine a corresponding action depending on the presence or absence of data loss, the presence of a defective area in the camera image where the ground was not captured due to water droplets, condensation, or dirt on the lens when the camera image was acquired, or clouds or fog, etc.

[0196] Next, the determination processing unit 2351 determines a corresponding action based on the state of the aircraft (step 302). For example, as shown in Fig. 9, the determination processing unit 2351 can determine a corresponding action based on state quantities such as the position, flight speed, flight altitude, and attitude of the aircraft.

[0197] Next, the determination processing unit 2351 determines a corresponding action based on the state of the sensor (step 303). For example, as shown in Fig. 9, the corresponding action can be determined according to the sensor settings such as the resolution, focus, and brightness of the camera.

[0198] Next, the response action determination result display unit 2352 displays the results of the response action determination made in steps 301 to 303 (step 304). For example, the inappropriate data, the reason for the inappropriate determination, and the response action for the inappropriate data, as shown in Fig. 13, are displayed.

[0199] Next, the user input accepting unit 2370 accepts a confirmation input from the user for the displayed response action determination result (step 305). That is, by operating a response action confirmation button as shown in Fig. 13, a confirmation input for the response action determined by the response action determination is accepted. Alternatively, an input of a response action other than the response action determination result is accepted from the user.

[0200] Next, the corresponding action determination unit 2350 determines the corresponding action based on the input information acquired from the user in step 305, and records the determined corresponding action (step 306).

[0201] Note that, although Figure 23 describes the processing flow of the response action determination unit 2350 of the data management system 2300, the processing flow of the response action determination unit 3340 of the data processing management unit 3300 can be a processing flow in which steps 302 and 303 in Figure 23 are omitted.

[0202] (A-1-11. Processing Flow of Appropriateness Judgment in Data Processing Management Unit 3300) FIG. 24 is a flowchart showing the processing flow performed when the data appropriateness judgment unit 3330 in the data processing management unit of the present invention judges the appropriateness of data.

[0203] First, the data analysis unit 3331 judges the suitability of the data being processed, obtained from the data processing unit 3200, based on its contents (step 401).

[0204] Next, the data analysis unit 3331 judges the suitability of the processed data obtained from the data processing unit 3200 based on the data content (step 402).

[0205] Next, the data appropriateness determination unit 3330 determines whether or not there was an inappropriate determination as a result of the appropriateness determinations made in steps 401 and 402, and determines the process to transition to based on the determination result (step 403). If there is an inappropriate determination, the process transitions to step 404, and if there is no inappropriate determination, the process transitions to step 406.

[0206] Next, the inappropriate data that has been judged to be inappropriate and the details of the inappropriateness judgment are displayed by the appropriateness judgment result display unit 3332 (step 404). For example, the inappropriate data as shown in Fig. 12, the reason for the inappropriateness judgment for the inappropriate data, the degree of exceedance of the reference value, the sensing time, the location of the target ground area, etc. are displayed.

[0207] Next, the user input receiving unit 3360 receives a confirmation input from the user for the displayed inappropriateness determination result (step 405). That is, by operating the confirmation button for inappropriate data as shown in Fig. 12, the inappropriateness determination result is confirmed for the data in process or the processed data that has been determined to be inappropriate by the inappropriateness determination and the contents of the determination.

[0208] Next, regardless of the result of the suitability judgment, the display device displays to the user the data in process and the processed data that have not been judged to be inappropriate in the data suitability judgment (step 406).

[0209] Next, the user input receiving unit 3360 receives a selection input from the user of the data in process or the processed data that should be judged as inappropriate (step 407). Even for data that has not been judged as inappropriate in steps 401 and 402, the user can manually judge it as inappropriate and input it into the system.

[0210] Next, the data appropriateness determination unit 3330 determines the inappropriate data and the content of the inappropriateness determination based on the input information acquired from the user in step 407, and records the determined inappropriate data and the content of the inappropriateness determination (step 408).

[0211] (A-1-12. Processing flow for suitability determination when SfM processing is performed in the data processing unit 3200) Fig. 25 is a flowchart showing the processing flow for data processing and data suitability determination when SfM processing is performed by the data processing center system of the present invention. Using this diagram, each processing step when SfM processing is performed in the data processing unit 3200 and an example of the data suitability determination unit 3330 determining suitability based on multiple pieces of data being processed will be described.

[0212] First, the data processing unit 3200 detects feature points from an image and estimates the three-dimensional distribution of the feature points (step 501).

[0213] Next, the data appropriateness determination unit 3330 determines whether or not feature points were extracted in step 501, and determines the process to transition to based on the determination result (step 502). If feature points were extracted, the process transitions to step 503, and if feature points were not extracted, the process transitions to step 512, where the data is determined to be inappropriate.

[0214] Next, the data processing unit 3200 determines tie points, which are points that can be recognized from both of the multiple images, from the image feature points (step 503).

[0215] Next, the data processing unit 3200 estimates the position and attitude at the time of photographing by the aircraft as exterior orientation parameters (step 504).

[0216] Next, the data processing unit 3200 estimates the lens distortion as an internal orientation parameter (step 505).

[0217] Next, the data appropriateness determination unit 3330 determines whether the parameters (aperture (f-number), focus amount, etc.) used to estimate the lens distortion in step 505 have changed, and determines the process to transition to based on the determination result (step 506). If the parameters have not changed, the process transitions to step 507, and if the parameters have changed, the process transitions to step 512, where it is determined that the data is inappropriate.

[0218] Next, the data processing unit 3200 detects ground control points (GCPs) and accuracy verification points (CPs) (step 507).

[0219] Next, the data suitability determination unit 3330 determines whether or not ground control points (GCPs) and accuracy verification points (CPs) were detected in the processing step of step 507, and determines the processing to transition to based on the determination result (step 508). If ground control points (GCPs) and accuracy verification points (CPs) were detected, the process transitions to step 509, and if ground control points (GCPs) and accuracy verification points (CPs) could not be detected, the process transitions to step 512, where it is determined that the data is inappropriate.

[0220] Next, the data processing unit 3200 performs a doming countermeasure process (step 509). The doming countermeasure process will be described in detail later.

[0221] Next, the data processing unit 3200 constructs a point cloud model from the image feature points and a plurality of stereo pair images (step 510).

[0222] Next, the data appropriateness determining unit 3330 determines whether the data is appropriate (step 510).

[0223] (A-1-13. Explanation of doming countermeasure processing) The concept of the doming countermeasure processing executed in step 509 of the flowchart shown in Fig. 25 will be briefly explained using Fig. 26. Fig. 26 is a diagram showing an example of the doming countermeasure processing executed by the data processing unit of the present invention.

[0224] As shown in the upper diagram of Fig. 26, doming in this embodiment refers to an error that occurs when SfM processing is performed using multiple nadir images taken from the sky toward the ground, causing the shape of the ground surface estimated by SfM processing to be more curved than the actual shape of the ground surface. Such doming occurs when SfM processing is performed using only nadir images or when there are errors in the interior orientation parameters.

[0225] As a countermeasure against doming that takes such doming into account, SfM processing is performed using multiple images, including a nadir image and an image facing diagonally downward, thereby reducing the estimation error of the ground surface shape due to doming, as shown in the lower diagram of Figure 26.

[0226] In each of the above-described embodiments, the data management system 2300 is implemented within the data acquisition base system 2000, but all or part of the functions of the data management system 2300 can also be implemented in the aircraft 1000, the data processing base system 3000, the distribution system 4000, or other systems.

[0227] The above-described embodiments are merely examples for facilitating understanding of the present invention, and are not intended to limit the present invention. The present invention can be modified and improved without departing from the spirit thereof, and it goes without saying that the present invention includes equivalents thereof.

[0228] [A-2. Effect of this embodiment] In the above-described embodiment, particularly when the data suitability determination unit determines that sensing data is inappropriate, accompanying information including the content of the suitability determination, the reason for the suitability determination, corresponding actions, etc. is displayed to the user (data processing staff) of the data processing management unit. Therefore, instead of immediately deciding not to use the inappropriate data, the data processing staff can perform data processing that allows the use of inappropriate data, thereby making it possible to effectively use the inappropriate data.

[0229] Furthermore, by displaying accompanying information in association with the processed data or corresponding information such as the ground location or aircraft location, a viewing user such as a crisis management headquarters who views the processed data can correctly recognize the locations where data determined to be a shooting failure is being used and the locations where it is not, and the reason why inappropriate data was determined to be inappropriate, and can also grasp the range of applications in which the processed data can be used, allowing a correct understanding of the situation in the target area based on the processed data.Furthermore, since it is possible to grasp future response actions, such as whether the inappropriate data is scheduled to be updated by re-measurement, if a user who checks the processed data is a member of a crisis management headquarters or the like, they can more appropriately determine plans for future crisis response work.

[0230] Furthermore, for users of the data management system 2300, for example, persons in charge of sensing work using aircraft, accompanying information including the content of the judgment on the suitability of the sensing data, the reason for the judgment, and corresponding actions can be displayed, so that the persons in charge of sensing work can accurately understand whether or not it is necessary to re-acquire data for data that is judged to have failed to be captured, and if it is necessary to redo the sensing, the sensing process can be quickly carried out again, and the processed data based on the re-sensing can be provided to viewing users more quickly.

[0231] 1...Aviation data processing system 100...Input device 200...Output device 300...Processing device 400...Main memory device 500...Auxiliary memory device 600...Communication device 700...Bus 1000...Aircraft 1100...Flight unit 1110...Self-position / speed determination unit 1120...Attitude determination unit 1130...Flight control unit 1200...Sensing unit 1210...Sensor 1220...Sensor attitude control unit 1230...Sensor control unit 1300...Communication unit 1310...Control communication unit 1320...Data communication unit 1330...Status data communication unit 1400...Status determination unit 1410...Flight state determination unit 1420...Sensing state determination unit 1430...Communication state determination unit 1500...Sensing data recording unit 1600...Status data recording unit 1610...Flight control information recording unit 1620... Measurement control information recording unit 1630... Status determination result recording unit 2000... Data acquisition base system 2100... Communication infrastructure management system 2200... Aircraft flight operation system2300...Data management system 2310...User request acquisition unit 2320...Sensing data acquisition unit 2330...Status information acquisition unit 2340, 3330...Data suitability determination unit 2341, 3331...Data analysis unit 2342...Status information analysis unit 2343, 3332...Suitability determination result display unit 2350, 3340...Response action determination unit 2351, 3341...Determination processing unit 2352, 3342...Response action determination result display unit 2360, 3350...Associated information assignment unit 2361, 3351...Associated information generation unit 2362, 3352...Information assignment unit 2363, 3310...Associated information display unit 2370, 3360...User input acceptance unit 2500...Flight operation management system 2600...Airspace monitoring system 3000...Data processing base system 3100...Communication section 3200...Data processing section 3300...Data processing management section 3320...Data processing execution command section 3370...Distribution information generation section 4000...Distribution system 4100...Communication section 4200...Distribution data management section 5000...User terminal device

Claims

1. A control system comprising: a data acquisition unit that acquires at least any of the following data to be judged: sensing data including images or point cloud information of a target area acquired using a sensor mounted on an aircraft, data in process generated in the course of data processing of the sensing data, and processed data generated by data processing of the sensing data; a data suitability judgment unit that judges the suitability of the data to be judged acquired by the data acquisition unit; and an associated information generation unit that generates associated information including information regarding the content of the suitability judgment when the data is judged to be inappropriate by the data suitability judgment unit, and the control system records in a recording unit or displays on a display unit the inappropriate data judged to be inappropriate, or the ground position of the target area related to the inappropriate judgment, or the position of the aircraft, or the control target position of the aircraft, or the control target route of the aircraft, in association with the associated information.

2. A control system according to claim 1, wherein the associated information includes information regarding the reason why the suitability judgment determined the device to be inappropriate.

3. A control system as described in claim 1, wherein the associated information is displayed on at least one of a screen in the form of a map, a three-dimensional model, or a three-dimensional space in association with the inappropriate data, or the ground position of the target status related to the inappropriate judgment, or the position of the aircraft, or the control target position of the aircraft, or the control target route of the aircraft.

4. A control system as described in claim 1, wherein the associated information includes information on the degree to which the inappropriate data exceeds a standard value for suitability judgment when the inappropriate data is judged to be inappropriate by the suitability judgment.

5. A control system as described in claim 1, wherein the associated information includes at least one of the sensing date and time regarding the acquisition of the sensing data, the aircraft identification information, the sensor identification information, the aircraft control information, the sensor control information, and status information of the sensing data.

6. A control system as described in claim 1, wherein the data suitability judgment unit uses preset judgment conditions or artificial intelligence to judge the suitability of the data to be judged, or displays candidates for inappropriate data.

7. A control system according to claim 1, wherein the data suitability judgment unit judges the suitability of the data to be judged based on suitability judgment input information for the data to be judged received from a user.

8. A control system as described in claim 6, wherein, when the data suitability judgment unit judges the suitability of the sensing data, the judgment conditions include at least one of the state of the sensing data, the state of the flying object, and the state of the sensor, or, when the data suitability judgment unit judges the suitability of the processed data, the judgment conditions include the state of the processed data.

9. A control system as described in claim 8, wherein when the data suitability judgment unit judges the suitability of the sensing data and the judgment condition includes the state of the flying object, the judgment condition is set by comparing the state quantity related to the flying object with a reference value, or a comparison with a past actual value, or a comparison with a control target value of the state quantity to be controlled related to the flying object.

10. A control system as described in claim 8, wherein when the data suitability judgment unit judges the suitability of the sensing data and when the judgment conditions include the state of the sensing data, the judgment conditions are set by comparing a state quantity related to the sensing data with a reference value, or a comparison with a state quantity of past sensing data, or a comparison with a target value, or the acquisition rate of sensing data for a target area to be sensed.

11. A control system as described in claim 8, wherein the judgment criteria when the data suitability judgment unit judges the suitability of the data being processed or the processed data are set by comparing the state quantity related to the data being processed or the processed data with a reference value, or by comparing the state quantity of past processed data, or by comparing with a target value.

12. A control system as described in claim 1, further comprising a response action determination unit that determines a response action to be taken when the data to be judged is judged to be inappropriate by the data appropriateness determination unit, and executes the response action.

13. A control system as described in claim 12, wherein the corresponding action determination unit determines the corresponding action based on at least one of the contents of the data to be determined, the state of the flying object, the state of the sensor, and the result of the suitability determination.

14. A control system according to claim 12, wherein the response action determination unit determines the response action according to designated input information for the response action received from a user.

15. A control system as described in claim 13 or 14, wherein the response action includes at least any of the following: not using the inappropriate data, substituting other data for the inappropriate data, notifying the user of restrictions on use of the processed data, notifying the user that the processed data contains inappropriate data, specifying a data processing method, re-performing sensing using an aircraft, specifying a data processing method according to a user-specified use, and specifying a display method for the processed data according to a user-specified use.

16. A control system according to claim 12, wherein the associated information includes information regarding the corresponding action.

17. A control system as described in claim 16, wherein when the information on the response action included in the associated information is non-use of the inappropriate data, information on a target area related to the inappropriate data is included in the associated information.

18. A control system as described in claim 16, wherein, when the information on the response action included in the associated information is a substitution of other data in place of the inappropriate data, information on the other data to be substituted is included in the associated information.

19. A control system as described in claim 16, wherein when the information on the corresponding action included in the associated information is a notification of a restriction on the usage of the processed data, information on the restriction on the usage is included in the associated information.

20. A control system as described in claim 16, wherein, when the information on the corresponding action included in the associated information is a designation of the data processing method, information on the designated data processing method is included in the associated information.

21. A control system as described in claim 16, wherein when the information on the response action included in the associated information is the re-execution of sensing by the flying object, the associated information includes sensing re-execution schedule information including at least one of the planned location, planned area, and planned time for the sensing re-execution.

22. A control system as described in claim 12, wherein the control system receives from a user usability determination conditions for determining whether the inappropriate data is usable by at least one of the response actions of notifying the user of restrictions on the use of the processed data, notifying the user that the processed data contains inappropriate data, and specifying a data processing method, and when the inappropriate data is determined to be inappropriate by the suitability determination, determines whether the inappropriate data is usable or displays the usable inappropriate data in accordance with the usability determination conditions.

23. A control system as described in claim 12, which receives from a user usage judgment conditions for determining that the inappropriate data can be used by the response action, such as at least one of notification of restrictions on use of the processed data, notification that the processed data contains inappropriate data, and specification of a data processing method, and when the inappropriate data is judged to be inappropriate by the suitability judgment, determines a response action using the inappropriate data in accordance with the usage judgment conditions, or displays candidate response actions using the inappropriate data.

24. A control system as described in claim 12, wherein when the suitability judgment results in an inappropriate response, the control system judges a response action or displays a response action according to judgment conditions for the response action received from a user or by utilizing artificial intelligence.

25. A control method in which a computer executes the following steps: a data acquisition step for acquiring data to be judged, which is at least any of sensing data including images or point cloud information of a target area acquired using a sensor mounted on an aircraft, in-process data generated in the course of data processing of the sensing data, and processed data generated by data processing of the sensing data; a data suitability determination step for making a suitability determination for the data to be judged acquired by the data acquisition unit; an associated information generation step for generating associated information including information regarding the content of the suitability determination for inappropriate data judged to be inappropriate; and a step of associating the inappropriate data, or the ground position of the target area corresponding to the inappropriate data, or the position of the aircraft, or the control target position of the aircraft, or the control target route of the aircraft, with the associated information, and recording it in a recording unit or displaying it on a display unit.

26. A control program that causes a computer to execute the following steps: a data acquisition step of acquiring at least any of the data to be judged, which are sensing data including images or point cloud information of a target area acquired using a sensor mounted on an aircraft, data in process generated in the process of data processing of the sensing data, and processed data generated by data processing of the sensing data; a data suitability judgment step of making a suitability judgment on the data to be judged acquired by the data acquisition unit; an associated information generation step of generating associated information including information regarding the content of the suitability judgment for inappropriate data judged to be inappropriate; and a step of associating the inappropriate data, or the ground position of the target area corresponding to the inappropriate data, or the position of the aircraft, or the control target position of the aircraft, or the control target route of the aircraft, with the associated information, and recording it in a recording unit or displaying it on a display unit.

Images