Data processing device and data processing method

Abstract

An object is to obtain a data processing device that can accurately align the observed position corresponding to data obtained by multiple sensors. The data processing device (1) according to the present invention includes: feature element extraction units (14, 22) that extract feature elements, which are parts corresponding to reference objects in the object area, from map information that is data of a map including the object area and multiple observation data obtained by respectively observing the object area with multiple sensors; a correspondence association unit (16) that selects one of the map information and the multiple observation data as reference data to be used as a reference for alignment; and an alignment unit (23) that aligns each of the multiple observation data so that the position of the feature element corresponding to the observation data is consistent with the position of the feature element in the reference data.

Description

Technical Field

[0001] The present invention relates to a data processing device and a data processing method for processing information obtained from sensors. Background Art

[0002] In recent years, sensor information has been effectively used in various fields. Sensor information is information obtained through observations by sensors, which is data acquired by sensors or data obtained by processing such data. Examples of sensors include passive sensors that passively observe electromagnetic waves and active sensors that transmit electromagnetic waves and observe the reflected electromagnetic waves. Examples of passive sensors include so-called image sensors, and examples of active sensors include synthetic aperture radar (SAR) and laser scanners.

[0003] Sensor information can be used, for example, to grasp the situation of natural disasters such as earthquakes and heavy rains or man-made disasters such as fires and accidents. In addition, not only for grasping the situation of disasters, but also research is being conducted on using sensor information to detect disaster omens to contribute to disaster prevention measures. Furthermore, sensor information is also used for grasping the growth state of crops, the situation of traffic jams, and the like.

[0004] Patent Document 1 discloses a technique for grasping areas where liquefaction, embankment collapse, etc. occur using images obtained by observations of SAR mounted on artificial satellites. The ground deformation visualization device described in Patent Document 1 uses data acquired by SAR at different times to generate an interference fringe image and an image representing a region with low interferometry, and displays these images overlapped. Areas where the ground surface has changed due to liquefaction, embankment collapse, etc. are displayed as regions with low interferometry, but in rivers, etc., the state of the water surface always changes regardless of these phenomena, so the interferometry is inherently low. In Patent Document 1, the interference fringe image and the image representing the region with low interferometry are displayed overlapped, so it is easy for the user to determine whether the region with low interferometry is a river or the like, or an area where liquefaction, embankment collapse, etc. have occurred. In addition, Patent Document 1 also describes further displaying these images overlapped with a topographic representation map.

[0005] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2018-40728 Summary of the Invention

[0006] Problems to be Solved by the Invention

[0007] However, according to the technology described in the above-mentioned Patent Document 1, when an image obtained by SAR observation is displayed overlapping with a topographic representation map, alignment is performed based on the position of the ground surface corresponding to the observation data obtained in the processing of the SAR observation data and the map coordinates in the topographic representation map. Generally, information such as the orbital information and attitude information of a satellite, and the off-nadir angle are used to calculate the position of the ground surface corresponding to the observation data of the SAR mounted on the satellite. However, the position calculated in this way may not necessarily match the map coordinates due to differences in the definition of the coordinate system, errors in the information used in the calculation, etc. That is, the coordinate system represented by the position obtained in the processing of the SAR observation data may be different from the coordinate system of the map coordinates. Therefore, in the technology described in Patent Document 1, if the topographic representation map and an image representing a region with low interference are displayed overlappingly, the user may misidentify the position of the region with low interference.

[0008] On the other hand, in recent years, various types of sensors have been installed in various places, and sometimes these sensors observe the same part. Since the characteristics of the data obtained vary depending on the type of sensor or the installation location, etc., if the observation data of multiple sensors that observe the same part are used, the information related to that part increases. However, since the positions of these data are calculated separately, if the data obtained by these multiple sensors are displayed overlappingly as they are, for example, as in the technology described in the above-mentioned Patent Document 1, there may be a position shift. In addition, not limited to display, in the case of performing analysis such as detecting disaster sites, grasping the growth state of crops, and grasping the traffic congestion situation using the data obtained by these multiple sensors, if alignment is not performed with high precision, a highly accurate analysis result cannot be obtained.

[0009] The present invention has been completed in view of the above circumstances, and an object thereof is to obtain a data processing device capable of performing alignment of the observed positions corresponding to the data obtained by multiple sensors with high precision.

[0010] Solution to the problem

[0011] In order to solve the above problems and achieve the object, the data processing device according to the present invention includes a feature element extraction unit that extracts feature elements corresponding to a reference object in an object area from map information that is data of a map including the object area and a plurality of observation data obtained by observing the object area by a plurality of sensors respectively. The data processing device further includes: a selection unit that selects one of the map information and the plurality of observation data as reference data to be used as a registration reference; and a registration unit that registers each of the plurality of observation data such that the position of the feature element corresponding to the observation data coincides with the position of the feature element in the reference data. The feature element extraction unit generates, for each sensor, simulated feature elements that simulate the data obtained by observing the reference object by the sensor using the feature elements extracted from the map information, and extracts feature elements from the observation data corresponding to the sensor using the corresponding simulated feature elements for each sensor.

[0012] Effect of the Invention

[0013] The data processing device according to the present invention has an effect of being able to accurately register the observed positions corresponding to the data obtained by a plurality of sensors. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 FIG. is a diagram showing a structural example of the data processing device according to Embodiment 1.

[0015] Figure 2 FIG. is a diagram showing a structural example of a computer system that implements the data processing device of Embodiment 1.

[0016] Figure 3 FIG. is a diagram showing an example of the overall processing flow in the data processing device of Embodiment 1.

[0017] Figure 4 FIG. is a diagram showing a structural example of the sensor information of Embodiment 1.

[0018] Figure 5 FIG. is a flowchart showing an example of the corresponding association processing procedure of the feature elements of Embodiment 1.

[0019] Figure 6 FIG. is a diagram showing an example of the characteristic information of Embodiment 1.

[0020] Figure 7 FIG. is a diagram showing an example of a display screen in which the change area extracted by the analysis unit of Embodiment 1 is overlaid on the map information.

[0021] Figure 8This is a diagram showing an example of overlapping the displacement amount calculated by the analysis unit of Embodiment 1 on the display screen of map information.

[0022] Figure 9 This is a diagram showing an example of overlapping the flooded area calculated by the analysis unit of Embodiment 1 on the display screen of visual sensor image data.

[0023] Figure 10 This is a diagram showing a structural example of the data processing apparatus of Embodiment 2.

[0024] Figure 11 This is a diagram showing an example of the overall processing flow in the data processing apparatus of Embodiment 2.

[0025] Figure 12 This is a flowchart showing an example of the characteristic element correspondence association process of the correspondence association unit of Embodiment 2.

[0026] Figure 13 This is a diagram showing a structural example of the data processing apparatus of Embodiment 3.

[0027] Figure 14 This is a flowchart showing an example of the simulated characteristic element generation process of Embodiment 3.

[0028] Figure 15 This is a diagram showing an example of the overall processing flow in the data processing apparatus of Embodiment 3.

[0029] (Explanation of reference numerals)

[0030] 1, 1a, 1b: Data processing apparatus; 2: Sensor information providing apparatus; 3: Map information providing apparatus; 11: Sensor information acquisition unit; 12: Map information acquisition unit; 13: Processing information storage unit; 14, 14a, 22: Characteristic element extraction unit; 15-1 to 15-n: First processing unit to nth processing unit; 16, 16a, 16b: Correspondence association unit; 17: Analysis unit; 18: Display processing unit; 19: Selection unit; 21: Calibration unit; 23: Alignment unit; 31: Information output unit; 32: Adjustment unit. Detailed description of the specific embodiment

[0031] Hereinafter, the data processing apparatus and the data processing method according to the embodiments of the present invention will be described in detail with reference to the drawings. In addition, the present invention is not limited by this embodiment.

[0032] Embodiment 1.

[0033] Figure 1FIG. 0 is a diagram showing a structural example of a data processing apparatus according to Embodiment 1 of the present invention. The data processing apparatus 1 of the present embodiment is an apparatus that acquires sensor information corresponding to a plurality of sensors from a sensor information providing apparatus 2 and aligns the sensor information. The aligned sensor information is used, for example, for extracting a disaster site in the event of a disaster, grasping the growth state of crops, grasping the traffic jam situation, etc. In addition, if sensor information at a plurality of time points is used, their displacement can also be grasped.

[0034] The sensor information includes observation data obtained by observation of the sensor. The observation data is data acquired by the sensor or data obtained by processing the data. In addition, the sensor information includes information indicating the observation date and time of the sensor and observation information for calculating the position of the observation site observed by the sensor. The sensor may be a sensor mounted on a satellite, aircraft, drone, helicopter, etc., a sensor mounted on a vehicle, etc., or a sensor fixed to the ground surface, building, etc.

[0035] The sensor may be a passive sensor or an active sensor. As a passive sensor, an image sensor that observes visible light, infrared light, ultraviolet light, etc. is exemplified. As an active sensor, SAR, LIDAR (Laser Imaging Detection and Ranging) that acquires three-dimensional point cloud data is exemplified. The sensor may also be a sensor other than these.

[0036] The data processing apparatus 1 is connected to a sensor information providing apparatus 2 that provides sensor information and a map information providing apparatus 3 that provides map information via communication lines, respectively. These communication lines may be wired lines, wireless lines, or a mixture of wireless lines and wired lines. Here, an example in which the data processing apparatus 1 acquires information from the sensor information providing apparatus 2 and the map information providing apparatus 3 via the communication lines is described, but the method by which the data processing apparatus 1 acquires information from the sensor information providing apparatus 2 and the map information providing apparatus 3 is not limited to the example via the communication lines. For example, the sensor information providing apparatus 2 may record the sensor information in a recording medium, and the data processing apparatus 1 may read the information from the recording medium. Similarly for the map information, it may also be provided from the map information providing apparatus 3 to the data processing apparatus 1 via the recording medium.

[0037] In addition, the sensor information providing device 2 and the map information providing device 3 can be either external devices managed by a manager other than the organization that operates and manages the data processing device 1, or devices managed by the organization that operates and manages the data processing device 1. For example, when the data processing device 1 provides the aligned sensor information or the analysis result using the aligned sensor information to a local government for disaster countermeasures or the like, the sensor information providing device 2 and the map information providing device 3 are managed by the organization that operates and manages the data processing device 1, and acquire wide-area sensor information and status information of the whole country or the like from external devices. Moreover, the data processing device 1 can also acquire corresponding information from the sensor information providing device 2 and the map information providing device 3 for each local government.

[0038] As Figure 1 shown, the data processing device 1 includes a sensor information acquisition unit 11, a map information acquisition unit 12, a processing information storage unit 13, a feature element extraction unit 14, first processing units 15-1 to nth processing units 15-n, a corresponding association unit 16, an analysis unit 17, a display processing unit 18, an information output unit 31, and an adjustment unit 32. n is the number of sensors corresponding to the sensor information to be processed by the data processing device 1. For example, when the data processing device 1 processes the sensor information corresponding to four sensors, namely, an infrared sensor as an infrared image sensor, a visible sensor as a visible image sensor, SAR, and LIDAR, n is 4. In addition, n is the maximum number of sensors to be processed, and the data processing device 1 can also perform processing using the sensor information corresponding to a part of the n sensors. For example, the data processing device 1 that processes the four sensors of infrared sensor, visible sensor, SAR, and LIDAR can also perform processing using the sensor information of the infrared sensor, visible sensor, and SAR without using the sensor information of LIDAR.

[0039] The sensor information acquisition unit 11 acquires sensor information from the sensor information providing device 2. The sensor information acquisition unit 11 stores the acquired sensor information in the processing information storage unit 13. When the sensor is on a satellite, the sensor information providing device 2 provides the data observed by the sensor or the data obtained by processing the data together with the observation information indicating the orbital position of the satellite, the operating conditions of the sensor, etc. When the sensor is on a satellite, the sensor information acquisition unit 11 acquires the sensor information including the target area to be analyzed from the sensor information providing device 2. The target area is appropriately set according to the content of the analysis performed by the data processing device 1. When the data processing device 1 performs an analysis for assisting the disaster response of a local government, the target area is the area including the local government. When the data processing device 1 performs a process with the entire Earth as the analysis target, the target area is the entire Earth.

[0040] The sensor information acquisition unit 11 acquires sensor information, for example, according to the determined data size. The determined data size is, for example, the minimum data unit when the sensor information providing device 2 distributes sensor information, but is not limited thereto. In addition, the sensor information acquisition unit 11 may acquire sensor information according to the size of the determined observation area instead of acquiring sensor information according to the determined data size. Hereinafter, a set of sensor information determined according to the data size or the size of the observation area is also referred to as sensor information corresponding to one image. When the target area is larger than the area corresponding to the sensor information corresponding to one image, the sensor information acquisition unit 11 acquires sensor information corresponding to multiple images.

[0041] In addition, when the data processing device 1 performs an analysis of extracting a change area by using the difference between the sensor information before and after a disaster as described later, in normal times when no disaster has occurred, the sensor information corresponding to the target area is acquired at least once in advance and the alignment process described later is performed. Or, when the sensor information providing device 2 also provides past sensor information, the sensor information acquisition unit 11 may, when determining the priority order of response after a disaster, acquire the past sensor information, that is, the sensor information before the disaster, together with the sensor information after the disaster and perform the alignment process.

[0042] Observation data of artificial satellites are generally provided as data at various levels from raw data to high-order products. The sensor information of the present embodiment can correspond to data at any of these levels. In addition, in the present embodiment, in order to be used in the analysis in the analysis unit 17 or for display, image data showing the brightness, scattering intensity, altitude, etc. of each pixel corresponding to a position or three-dimensional point group data represented by points of three-dimensional coordinate values is used. Therefore, when the observation data provided as sensor information is not image data or the like, the first processing unit 15-1 to the nth processing unit 15-n corresponding to the sensor perform processing for generating image data from the observation data. In addition, when the sensor information includes observation data that has not been subjected to sensor-specific radiation calibration, geometric correction, etc., the first processing unit 15-1 to the nth processing unit 15-n corresponding to the sensor only need to perform sensor-specific radiation calibration, geometric correction, etc. to generate image data.

[0043] Regarding the sensor information of sensors mounted on aircraft, vehicles, etc., it is also obtained from the sensor information providing device 2 together with the position information in the same way. However, the observation data of sensors mounted on artificial satellites can be obtained regardless of whether a disaster occurs, while sensors mounted on aircraft, vehicles, etc. sometimes observe the area where a disaster is supposed to occur after the disaster. In this case, it may be impossible to obtain the sensor information before the disaster. In the case where the sensor information before the disaster cannot be obtained, the sensor information before the disaster can also be generated by simulation or the like. Or, for example, the sensors mounted on aircraft, vehicles, etc. can also be observed regularly in advance usually, and the sensor information acquisition unit 11 can obtain the usual sensor information from the sensor information providing device 2.

[0044] When the sensor is a sensor with a fixed position, the sensor information providing device 2 can also be the sensor itself or a processing device connected to the sensor, etc. For example, when the sensor is a roadside sensor, a camera installed in a building, etc., the sensor information acquisition unit 11 can also obtain the sensor information from the sensors in the observation target area. Or, when the sensor information of a plurality of fixed sensors is collected and managed by a server or the like, the server or the like becomes the sensor information providing device 2.

[0045] The map information acquisition unit 12 acquires map information, which is data of a map including the target area, from the map information providing device 3, and stores the acquired map information in the processing information storage unit 13. Specifically, the map information is, for example, three-dimensional map data, which is provided together with information indicating the accuracy of the three-dimensional map data. As the three-dimensional map data, there are various types of data, such as data generated based on observation data from sensors mounted on artificial satellites and covering the globe, and high-precision data generated for autonomous driving based on data observed by sensors such as LIDAR and cameras. The map information acquisition unit 12 can use either one type of three-dimensional map data or select map information with high accuracy from multiple types of three-dimensional map data according to the target area. The three-dimensional map data can be data represented by three-dimensional coordinate values such as a digital elevation model (DEM), or data in vector form.

[0046] The feature element extraction unit 14 extracts feature elements from the map information stored in the processing information storage unit 13. And, the feature element extraction unit 14 uses the extracted feature elements to generate, for each sensor, simulated feature elements that simulate the data obtained by observing the reference object by the sensor. Specifically, the feature element extraction unit 14 selects, from the map information of the target area of the area to be processed, the reference object on the ground used in the alignment. The reference object is at least one of a plurality of ground control points (GCPs), naturally formed structures, and buildings. Naturally formed structures are, for example, rivers, lakes, etc. Buildings are buildings, roads, railway tracks, etc. The ground control points are set for geometric correction and measurement, and their positions are determined with high accuracy. In addition, if linear structures such as rivers and roads are used as reference objects, the alignment accuracy can be improved. Also, if linear structures such as rivers and roads are used, even if the area of interest is the boundary of the imaging area, by considering the continuity of the linear feature objects across the boundary of the imaging area for alignment, the discontinuity at the boundary of the imaging area can be prevented. The feature element extraction unit 14 extracts the area corresponding to the selected reference object from the map information as the feature element. The feature element extraction unit 14 generates, based on the position information of the extracted feature element, the simulated data obtained when the reference object is observed by the sensor as the simulated feature element. When the sensor is an image sensor, the simulated data is image data, and when the sensor is LIDAR or the like, the simulated data is three-dimensional point cloud data. The position information of the feature element can be, for example, three-dimensional coordinate values representing the shape of the feature element, or information such as a polygon. Regarding the simulated feature element, imaging conditions such as the imaging direction, resolution, amplitude, and distortion are also reflected in its production.

[0047] The first processing unit 15-1 to the nth processing unit 15-n perform processing corresponding to n sensors respectively. The first processing unit 15-1 to the nth processing unit 15-n each include a calibration unit 21, a feature extraction unit 22, and an alignment unit 23. In addition, in Figure 1 , the calibration unit 21, the feature extraction unit 22, and the alignment unit 23 that constitute the first processing unit 15-1 are illustrated, but the second processing unit 15-2 to the nth processing unit 15-n also similarly include the calibration unit 21, the feature extraction unit 22, and the alignment unit 23. The calibration unit 21 of the first processing unit 15-1 to the nth processing unit 15-n reads the sensor information of the corresponding sensor from the sensor information stored in the processing information storage unit 13. Then, the calibration unit 21 of the first processing unit 15-1 to the nth processing unit 15-n performs sensor-specific calibration on the sensor information, including radiometric calibration, calculation of the position of the observation area, etc., thereby generating sensor image data. In addition, when the data processing device 1 acquires the sensor image data itself that has been calibrated such as radiometric calibration as sensor information, the corresponding first processing unit 15-1 to the nth processing unit 15-n may not perform the calibration process. Here, the sensor image data is not limited to general image data that is displayed as an image, but also includes data that can be displayed as an image by performing processing. An example of data that can be displayed as an image by performing processing is three-dimensional point cloud data. That is, the sensor image data of each sensor is obtained by the sensor observing the target area. In addition, the calibration unit 21 of the first processing unit 15-1 to the nth processing unit 15-n performs ortho calibration on the sensor image data of sensors mounted on artificial satellites, aircraft, drones, etc.

[0048] The feature extraction units 22 of the first processing unit 15-1 to the nth processing unit 15-n use the simulated feature elements generated by the feature extraction unit 14 to extract the regions corresponding to the reference object, i.e., feature elements, from the sensor image data, and calculate the similarity between the extracted feature elements and the simulated feature elements. Specifically, the feature extraction units 22 of the first processing unit 15-1 to the nth processing unit 15-n extract the region that most conforms to the shape represented by the simulated feature elements as the feature elements. Regarding the method of extracting the feature elements, methods generally used in image processing can be used, so detailed description is omitted. As the similarity, any index can be used, such as the Euclidean distance, correlation coefficient, etc. The feature extraction units 22 of the first processing unit 15-1 to the nth processing unit 15-n output the information indicating the position of the extracted feature elements together with the similarity to the correspondence association unit 16. In addition, in the present embodiment, the simulated feature elements of each sensor are generated based on the map data, and the simulated feature elements are compared with the feature elements extracted from the sensor image. However, it is not limited thereto, and the feature elements of the map data can also be simulated based on the feature elements of each sensor. In this case, the similarity between the feature elements corresponding to each sensor and the simulated data of the feature elements of the map data generated based on the sensor image data of each sensor is calculated.

[0049] In addition, the alignment units 23 of the first processing unit 15-1 to the nth processing unit 15-n generate local transformation parameters based on the coordinate transformation information described later output from the correspondence association unit 16. The local transformation parameters represent each element in the transformation matrix used to transform "the position of the feature element represented by the coordinate system corresponding to the sensor information, i.e., the coordinate value" into the coordinate value in the reference coordinate system. The alignment units 23 of the first processing unit 15-1 to the nth processing unit 15-n also perform the process of transforming the sensor image using the local transformation parameters, i.e., the alignment image transformation. The alignment units 23 of the first processing unit 15-1 to the nth processing unit 15-n save the sensor image data after the alignment image transformation as map matching information in the processing information storage unit 13. The details of the operations of the first processing unit 15-1 to the nth processing unit 15-n will be described later.

[0050] The corresponding association unit 16, which is the selection unit of the present embodiment, selects one of the map information and the multiple sensor information as the reference data to be used as the alignment reference. Specifically, the corresponding association unit 16 determines the reference coordinate system for the alignment of the multiple sensor information corresponding to the multiple sensors based on the similarity output from the first processing unit 15-1 to the nth processing unit 15-n and the characteristic information stored in the processing information storage unit 13. The reference coordinate system is the coordinate system of the map information or the sensor information selected as the reference data. The corresponding association unit 16 generates coordinate transformation information corresponding to each sensor based on the information indicating the positions of the feature elements extracted by the feature element extraction unit 14 and the information indicating the positions of the feature elements extracted by the first processing unit 15-1 to the nth processing unit 15-n. The corresponding association unit 16 outputs the coordinate transformation information corresponding to each sensor to the corresponding first processing unit 15-1 to the nth processing unit 15-n. The coordinate transformation information includes the coordinate values of the feature elements in the reference coordinate system and the coordinate values of the feature elements in the coordinate system corresponding to each sensor. The details of the operation of the corresponding association unit 16 will be described later.

[0051] The analysis unit 17 analyzes using the map matching information for each sensor stored in the processing information storage unit 13 and saves the analysis result in the processing information storage unit 13. An example of the analysis performed by the analysis unit 17 is a process of using sensor image data as observation data to extract the disaster-occurring area, and this process is, for example, a process of extracting the changed area that has changed before and after the disaster. The analysis unit 17, for example, uses the map matching information stored in the processing information storage unit 13 to extract the changed area on the surface of the earth and outputs the changed area information indicating the changed area as the analysis result. The surface of the earth includes the surface of the land, the sea surface, the water surfaces of lakes and rivers, and the buildings built on the ground surface. In addition, the analysis unit 17 can also extract the changed area of the atmosphere on the earth. The analysis unit 17, for example, extracts the changed area based on the difference between the map matching information before the disaster and the map matching information after the disaster. For example, the analysis unit 17 calculates the difference in brightness, scattering intensity, altitude, etc. of each pixel based on the data obtained by SAR in the map matching information and extracts the pixels with a difference above the threshold.

[0052] If the difference between consecutive pixels is above a threshold, the analysis unit 17 sets the portion corresponding to these consecutive pixels as one change region. In addition, the sensor information includes information on which pixel corresponds to which position. This information is constituted by, for example, the position of each pixel in each image and the information for calculating the position of each pixel based on this position. The information on which pixel corresponds to which position is not limited to this form and can be in any form. After the analysis unit 17 extracts the pixels with a difference above the threshold, it calculates the position corresponding to the pixel based on the above information, thereby being able to obtain the position of the change region. The analysis unit 17 outputs the position of the change region, that is, the position of the pixel with a difference above the threshold, as the analysis result. Generally, the change region includes a plurality of consecutive pixels. Therefore, when the difference varies according to the pixel, either the average value of the plurality of pixels can be set as the displacement amount, or the maximum value can be set as the displacement amount.

[0053] In addition, the analysis unit 17 can also, for each pixel, instead of the information observed before a disaster, which is used as a reference value for obtaining the change, that is, the difference, use the measurement results so far and the information calculated through analysis, etc. as reference information to obtain the difference. Here, an example of comparing brightness, scattering intensity, etc. for each pixel is shown, but it is not limited to this. When the altitude is included in the sensor information, the analysis unit 17 can also obtain the difference in altitude between the sensor information acquired in the past and the latest sensor information, and obtain the portion with a difference above the threshold as the change region. In addition, the analysis unit 17 can also extract the outlines of buildings, rivers, roads, etc. from both the map matching information obtained based on the sensor information acquired in the past and the map matching information obtained based on the latest sensor information through image processing, and obtain the difference in these outlines.

[0054] In addition, the analysis unit 17 can also perform analysis through machine learning that takes sensor image data as input, such as obtaining the threshold of the change region through machine learning.

[0055] In the above description, the changed area is extracted by comparing with certain reference information and calculating the difference. However, the extraction of the changed area may not use the reference information. For example, in the case where the sensor information is an infrared image obtained by a sensor that detects infrared light, it is possible to extract the area where the temperature corresponding to the pixel is above the threshold as the changed area caused by a fire or volcanic activity, etc. Not limited to infrared images, the analysis unit 17 can also extract the area where the brightness, scattering intensity, etc. corresponding to the pixel of the image obtained as the sensor information deviate from the pre-determined range as the changed area. Or, in the case where the area where the temperature is above the threshold is a fixed value or more, it is also possible to extract this area as the changed area. Or, it is also possible to extract the area where the temperature is higher than the average temperature of the surroundings by a fixed value or more as the changed area. In this way, the changed area is an area that has changed compared to the past state due to certain phenomena, or an area that has a difference from the surroundings. In addition, the calculation method of the changed area is not limited to these examples, and any commonly used method can be used.

[0056] In addition, the analysis unit 17 can also perform the process of extracting the flooded area after flooding as an analysis. And the analysis unit 17 can also perform the process of measuring the area of the flooded area. By measuring the area of the flooded area, it is possible to contribute to the grasp of the damage degree of the disaster. Similarly, the analysis unit 17 can also measure the area of the changed area.

[0057] The content of the analysis performed by the data processing device 1 is not limited to this. For example, it can also be the process of extracting the area where an abnormality that is expected to be involved in a disaster has occurred, the grasp of the growth condition of crops, the grasp of the traffic jam condition, the grasp of the number of vehicles in the parking lot of a commercial facility, etc. For example, the analysis unit 17 can also use the sensor information at multiple time points to analyze the change in the growth condition of crops, the change in the traffic jam condition, etc.

[0058] The process of extracting the area where an abnormality that is expected to be involved in a disaster has occurred is the same as the extraction of the above-mentioned changed area. For example, it is the process of extracting the area where the difference between the past sensor information as a reference and the newly acquired sensor information is above the threshold as the area where the abnormality has occurred. The process for grasping the growth condition of crops is, for example, the process of judging the growth of crops by obtaining the color by using sensor information obtained by imaging with multiple wavelengths of visible light. The process for grasping the traffic jam condition and the process for grasping the number of vehicles in the parking lot of a commercial facility are, for example, the process of extracting vehicles by performing image processing on the captured image and counting the number of vehicles. In addition, the specific content of the analysis is not limited to the above examples, and any processing method can be used. Hereinafter, as an example, the case where the content of the analysis is the process of extracting the changed area that has changed before and after the occurrence of a disaster will be mainly described.

[0059] In addition, when the data processing device 1 performs processing for monitoring the occurrence of a disaster or the occurrence of an abnormality involved in a disaster as an analysis, for areas that should be particularly vigilant, such as areas where linear precipitation bands are likely to occur, etc., they can also be set as the target areas for analysis to always obtain the latest sensor information and perform analysis. In addition, the data processing device 1 can also set the area where the disaster has occurred as the target area to obtain sensor information and perform analysis after the disaster has occurred. In addition, when applied to a disaster prevention system for a specific local government, etc., the area corresponding to the local government can also be set as the target area to always obtain the latest sensor information and perform analysis.

[0060] The display processing unit 18 overlays and displays the sensor image data obtained by transforming the alignment images of each sensor stored in the processing information storage unit 13. In addition, the display processing unit 18 can also further emphasize and display the analysis result of the analysis unit 17 on the sensor image data obtained by transforming the alignment images of each sensor stored in the processing information storage unit 13 and then overlay them. In this way, by overlaying the sensor image data obtained by transforming the alignment images, the display processing unit 18 can improve visibility. For example, although the image data obtained by SAR shows elevation, the colors of the ground surface, buildings, etc. are unknown. The display processing unit 18 can improve visibility by displaying these overlaid. In addition, the display processing unit 18 can improve visibility by emphasizing and displaying the analysis result.

[0061] The information output unit 31 reads out the information stored in the processing information storage unit 13 and outputs it to an external system or the like. For example, the information output unit 31 outputs the sensor image data after being transformed into a bit image, the analysis result, etc. to an external system. Thereby, the external system can display or process them. The adjustment unit 32 adjusts parameters and the like used in the processing such as the correction processing and the alignment processing in the data processing device 1 based on the input from the operator or the like, or based on the data received from other devices. For example, the data processing device 1 acquires information for correcting each sensor as observation information, but sometimes it is desired to change the parameters used in the correction in order to perform higher-precision correction. In such a case, by inputting to the data processing device 1 by the operator or the like, or by sending data from other devices, the change content is indicated to the data processing device 1. The adjustment unit 32 changes the parameters used in the correction based on the input from the operator or the like, or based on the data received from other devices. In addition, in the alignment processing, evaluation items, weights, evaluation formulas, etc. are determined as described later, but sometimes it is desired to change them. Similarly to the parameters used in the correction, the adjustment unit 32 changes the evaluation items, weights, evaluation formulas, etc. based on the input from the operator or the like, or based on the data received from other devices. In addition, after the operator or the like confirms the result of the alignment processing, the analysis result, etc., if it is determined that parameter adjustment is required based on these results, the change content may be indicated to the data processing device 1 as described above, and the adjustment unit 32 makes adjustments based on this. In addition, the adjustment unit 32 may also adjust the parameters when generating the simulated feature elements.

[0062] Next, the hardware structure of the data processing device 1 will be described. The data processing device 1 is implemented by a computer system. Figure 2 It is a diagram showing a structural example of the computer system that implements the data processing device 1 of the present embodiment. As Figure 2 shown, this computer system includes a control unit 101, an input unit 102, a storage unit 103, a display unit 104, a communication unit 105, and an output unit 106, which are connected via a system bus 107.

[0063] In Figure 2Among them, the control unit 101 is, for example, a CPU (Central Processing Unit), etc. The control unit 101 executes a data processing program that describes the data processing method of this embodiment. The input unit 102 is composed of, for example, a keyboard, a mouse, etc., and is used for the user of the computer system to input various information. The storage unit 103 includes various memories such as RAM (Random Access Memory) and ROM (Read Only Memory), and storage devices such as a hard disk, and stores the program that the control unit 101 should execute, the necessary data obtained during the processing, etc. In addition, the storage unit 103 is also used as a temporary storage area for the program. The display unit 104 is composed of an LCD (Liquid Crystal Display) panel, etc., and displays various screens for the user of the computer system. The communication unit 105 is a communication circuit that implements communication processing, etc. The communication unit 105 may also be composed of multiple communication circuits corresponding to multiple communication methods respectively. The output unit 106 is an output interface that outputs data to external devices such as a printer and an external storage device. In addition, Figure 2 is an example, and the structure of the computer system is not limited to Figure 2 the example of.

[0064] Here, an operation example of the computer system until it becomes a state where a data processing program capable of executing the processing of the data processing device 1 described in this embodiment is described. In a computer system having the above structure, for example, the data processing program is installed from a CD-ROM or a DVD-ROM placed in a CD (Compact Disc)-ROM drive or a DVD (Digital Versatile Disc)-ROM drive (not shown) into the storage unit 103. Then, when the data processing program is executed, the data processing program read from the storage unit 103 is saved in the area of the storage unit 103 that serves as the main storage device. In this state, the control unit 101 executes the processing of the data processing device 1 of this embodiment according to the first program saved in the storage unit 103.

[0065] In addition, in the above description, a CD-ROM or a DVD-ROM is used as the recording medium to provide the program that describes the processing in the data processing device 1, but it is not limited to this. Depending on the structure of the computer system, the capacity of the provided program, etc., for example, a program provided via the communication unit 105 using a transmission medium such as the Internet may be used.

[0066] Through Figure 2 the control unit 101 and the communication unit 105 shown to implement Figure 1 the sensor information acquisition unit 11 and the map information acquisition unit 12 shown. ThroughFigure 2 implemented by the control unit 101 shown Figure 1 the feature element extraction unit 14, the first processing unit 15-1 to the nth processing unit 15-n, the corresponding association unit 16, and the analysis unit 17 shown Figure 1 the processing information storage unit 13 shown is Figure 2 a part of the storage unit 103 shown. Through Figure 2 the control unit 101 and the display unit 104 shown to implement Figure 1 the display processing unit 18 shown. Through Figure 2 the communication unit 101 or the output unit 106 shown to implement Figure 1 the information output unit 31 shown. Through Figure 2 the input unit 102 and the control unit 101 shown to implement Figure 1 the adjustment unit 32 shown

[0067] Next, the details of the operation of this embodiment will be described Figure 3 is a diagram showing an example of the overall processing flow in the data processing device 1 of this embodiment. As Figure 3 shown, first, the acquisition of map information by the map information acquisition unit 12 (step S11) and the acquisition of sensor information corresponding to each sensor by the sensor information acquisition unit 11 (steps S21, S31, S41, S51) are performed. These information are stored in the processing information storage unit 13 as described above. In Figure 3 it shows an example where n, the number of sensors, is 4. In Figure 3 the example shown, the 4 sensors are SAR, visible sensor, infrared sensor, and LIDAR. SAR, visible sensor, and infrared sensor are mounted on the artificial satellite, and LIDAR is mounted on the vehicle. Hereinafter, the sensor information corresponding to SAR will also be referred to as SAR data, the sensor information corresponding to the visible sensor will also be referred to as visible sensor data, the sensor information corresponding to the infrared sensor will also be referred to as infrared sensor data, and the sensor information corresponding to LIDAR will also be referred to as LIDAR data

[0068] Hereinafter, as an example of the multiple sensors corresponding to the data processing device 1, these 4 sensors will be taken as an example for description, but the number of sensors, the types of sensors, and the mounting locations of the sensors are not limited to these. In addition, among the multiple sensors corresponding to the data processing device 1, there may also be included two or more sensors of the same type with different mounting positions or observation directions. Here, it is assumed that the first processing unit 15-1 performs the processing corresponding to SAR, the second processing unit 15-2 performs the processing corresponding to the visible sensor, the third processing unit 15-3 performs the processing corresponding to the infrared sensor, and the fourth processing unit 15-4 performs the processing corresponding to LIDAR

[0069] The calibration units 21 of the first processing unit 15-1 to the fourth processing unit 15-4 generate sensor image data by performing sensor-specific calibration, correction, etc. based on the sensor information stored in the processing information storage unit 13. When it corresponds to a sensor that requires orthorectification, orthorectification is performed on the sensor image data (steps S22, S32, S42, S52). Hereinafter, the sensor image data corresponding to SAR is also referred to as SAR image data, the sensor image data corresponding to the visible sensor is also referred to as visible image data, the sensor image data corresponding to the infrared sensor is also referred to as infrared image data, and the sensor image data corresponding to LIDAR is also referred to as three-dimensional point cloud data.

[0070] The sensor information includes observation information for calculating the observed position. The observation information includes, for example, information such as the position, orientation, and field-of-view angle range of the sensor. For example, when the sensor is mounted on a satellite, the observation information includes information such as the orbit information of the satellite, the attitude information of the satellite, and the nadir angle. Figure 4 It is a diagram showing a structural example of the sensor information of the present embodiment. In Figure 4 the example shown, the sensor information includes the imaging date and time, observation information, and sensor observation data. It is data obtained by the sensor or data obtained by processing this data. The imaging date and time is the observation date and time of the sensor. The observation information includes imaging position information and imaging angle information. The imaging position information is information indicating the position of the sensor, and the imaging angle information is information indicating the field-of-view direction of the sensor, that is, the orientation of the sensor. When the sensor is a sensor mounted on a moving body such as a satellite, an aircraft, a drone, or a vehicle, the imaging position information is information indicating the position of these moving bodies on which it is mounted. In a sensor mounted on a moving body, the imaging angle information may also include information related to the posture of the moving body and the field-of-view direction of the sensor based on the moving body.

[0071] Although not shown in the figure, the sensor information may also include other information required for sensor-specific calibration, correction, etc. in addition to the above-mentioned observation information. Alternatively, these other information may be provided separately from the sensor information, and the data processing device 1 acquires it separately from the sensor information. In addition, when the sensor is mounted on a moving body, the observation information may also include information indicating the accuracy of the position information of the moving body, the accuracy of the posture information of the moving body, and the accuracy of the field-of-view direction of the sensor. The content of the sensor information varies depending on the type of sensor and the mounting location, and is not limited to Figure 4 the example shown. The first processing unit 15-1 to the fourth processing unit 15-4 can perform sensor-specific calibration, correction, etc. based on the observation information and, if necessary, the above-mentioned other information to generate sensor image data.

[0072] In addition, in Figure 3 it is described that, regarding LIDAR, processing for generating three-dimensional point cloud data from sensor information, i.e., LIDAR data, is performed. However, when calibration or the like is required, calibration is also performed when generating the three-dimensional point cloud data.

[0073] SAR image data does not carry color. Although the resolution is lower compared to visual sensors, it can observe the ground surface even in the presence of clouds and can perform night observations. Additionally, by performing interferometry processing using the sensor information of SAR, the altitude can also be calculated. Moreover, although the observation range, i.e., the imaging range, of the image generated from the three-dimensional point cloud data is limited, a precise image can be obtained. Also, the altitude can be accurately calculated from the three-dimensional point cloud data. Additionally, infrared image data can be obtained even at night. Further, the temperature of the ground surface can be obtained from the infrared image data.

[0074] Thus, the characteristics differ depending on the type of sensor. Therefore, if the sensor information of multiple sensors is used, complementary results can be expected. For example, as described above, the characteristics of the images obtained by each sensor are different. Therefore, by displaying these images overlappingly, it is easier to grasp the state of the ground surface. However, since the observation methods, mounting locations, etc. of each sensor are different, in the case of only performing sensor-specific calibration, there may be a positional shift during overlapping. Additionally, for example, although the LIDAR mounted on a vehicle has high observation accuracy, the observation range is limited. Therefore, if the observation data of LIDAR is used to calibrate the observation data of SAR, which has a larger observation range than LIDAR, the altitude, etc. can be calculated with high accuracy over a wide range. However, when using the observation data of LIDAR to calibrate the observation data of SAR, which has a larger observation range than LIDAR, if the alignment accuracy between these data is low, the altitude, etc. with high accuracy cannot be calculated. Based on the above, it is desired to accurately align the sensor image data of multiple sensors.

[0075] Therefore, in the present embodiment, an object on the ground selected as a reference, i.e., a reference object, is chosen, and regions corresponding to the selected reference object are extracted from the map information and each sensor image data as feature elements. Then, it is regarded that the feature elements extracted from the map information and each sensor image data correspond to the same position, and alignment between sensors is performed. At this time, generally speaking, the accuracy of the position in the map information is high. Therefore, by performing alignment based on the map information as a reference, high-precision alignment can be carried out. On the other hand, although the map information has high accuracy when it is made, as time passes, the possibility of being different from the actual situation becomes higher due to building construction, occurrence of disasters, natural changes, etc. In contrast, the sensor can observe a state closer to the actual situation. Thus, it is also considered that depending on the conditions, the accuracy of the data obtained by the sensor is higher than that of the map information. Therefore, in the present embodiment, information expected to have high accuracy is selected from the map information and the sensor information corresponding to each sensor, and the coordinate system of this sensor information is set as the reference coordinate system, thereby achieving higher-precision alignment. As described below, the processing of steps S12, S23 to S25, S33 to S35, S43 to S45, and S53 to S55 shown in Figure 3 is performed.

[0076] Return to Figure 3 the description. The feature element extraction unit 14 generates simulated feature elements based on the map information stored in the processing information storage unit 13 (step S12). Specifically, as described above, the feature element extraction unit 14 extracts feature elements from the map information and generates simulated feature elements for each sensor based on the feature elements. The feature element extraction unit 14 outputs the generated simulated feature elements to the first processing unit 15-1 to the fourth processing unit 15-4 corresponding to the simulated feature elements. In addition, information indicating the position of the feature element is output to the correspondence association unit 16.

[0077] The feature extraction units 22 of the first processing unit 15-1 to the fourth processing unit 15-4 extract feature elements from the sensor image data using the analog feature elements received from the feature extraction unit 14 (steps S23, S33, S43, S53). Specifically, the feature extraction units 22 of the first processing unit 15-1 to the fourth processing unit 15-4 each use the analog feature elements received from the feature extraction unit 14 to extract, from the sensor image data, the region that most closely matches the shape represented by the analog feature elements as the feature elements. In addition, the feature extraction units 22 of the first processing unit 15-1 to the fourth processing unit 15-4 each calculate the similarity between the extracted feature elements and the analog feature elements, and output the information indicating the position of the extracted feature elements together with the similarity to the corresponding association unit 16. In addition, it is assumed that the information indicating the positions of the feature elements extracted by the feature extraction unit 14 and the information indicating the positions of the feature elements extracted by the feature extraction units 22 of the first processing unit 15-1 to the fourth processing unit 15-4 are represented by coordinate values with the same definition. When they are not represented by coordinate values with the same definition, the feature extraction units 22 of the first processing unit 15-1 to the fourth processing unit 15-4 perform coordinate transformation, for example, so that they become coordinate values with the same definition as those in the map information. In addition, the definition of the coordinate values mentioned here refers to definitions such as latitude, longitude, and altitude in the Japanese Geodetic System, or latitude, longitude, and altitude in the World Geodetic System, and three-dimensional orthogonal coordinates in the International Terrestrial Reference Coordinate System.

[0078] The corresponding association unit 16 performs corresponding association of the feature elements (step S61). Specifically, the corresponding association unit 16 determines the reference coordinate system in the alignment of the multiple sensor information corresponding to the multiple sensors based on the similarity received from each of the first processing unit 15-1 to the fourth processing unit 15-4 and the characteristic information stored in the processing information storage unit 13. The reference coordinate system is the coordinate system corresponding to the information estimated to be the most accurate in the map information and each sensor image data, and is determined in the process of step S61. Then, the corresponding association unit 16 generates coordinate transformation information corresponding to each sensor based on the analog feature elements of each sensor generated by the feature extraction unit 14 and the feature elements output from each of the first processing unit 15-1 to the fourth processing unit 15-4. The corresponding association unit 16 outputs the coordinate transformation information corresponding to each sensor to the corresponding first processing unit 15-1 to the fourth processing unit 15-4 respectively. The details of the operation of the corresponding association unit 16 will be described later.

[0079] The alignment units 23 of the first to fourth processing units 15-1 to 15-4 respectively generate local transformation parameters for transforming the coordinate values representing the positions of the feature elements in the sensor image data into the coordinate values in the reference coordinate system based on the coordinate transformation information received from the corresponding association unit 16 (steps S24, S34, S44, S54). The alignment units 23 of the first to fourth processing units 15-1 to 15-4 further use the local transformation parameters to perform a process of transforming the position of the sensor image into the coordinate values in the reference coordinate system, that is, alignment image transformation (steps S25, S35, S45, S55). Since the local transformation parameters are the transformation parameters obtained by performing alignment locally on the feature elements, when only the feature elements are transformed into the reference coordinate system using the local transformation parameters, alignment can be performed with high precision. On the other hand, for the parts other than the feature elements in the sensor image data, it may not be possible to perform alignment with high precision by performing coordinate transformation using the local transformation parameters. Especially in the case of a sensor mounted on a satellite, since the observation range is large, appropriate transformation parameters may vary depending on the position within one image. Therefore, the data processing device 1 may also use multiple feature elements within one image to obtain multiple local transformation parameters, and perform coordinate transformation on the image data between the feature elements by methods such as interpolation and use of the local transformation parameters. When it is assumed that the transformation parameters do not change within one image, the local transformation parameters may also be used for coordinate transformation of the sensor image data.

[0080] The alignment units 23 of the first to fourth processing units 15-1 to 15-4 respectively save the corresponding data after the alignment image transformation (steps S26, S36, S46, S56). Specifically, the first to fourth processing units 15-1 to 15-4 save the sensor image data after the alignment image transformation as map matching information in the processing information storage unit 13.

[0081] The display processing unit 18 performs visibility improvement processing (step S62) using the sensor image data obtained by transforming the alignment images of each sensor stored in the processing information storage unit 13. The visibility improvement processing is, for example, processing for generating display data for overlapping and displaying a plurality of sensor image data respectively corresponding to a plurality of sensors, or processing for generating display data for emphatically overlapping the analysis result stored in the processing information storage unit 13 on at least one of the plurality of sensor image data. The display processing unit 18 displays the display data generated by the visibility improvement processing, thereby performing overlapping display (step S64). Here, the overlapping display, as described above, represents, for example, the overlapping display of a plurality of sensor image data, or the overlapping display of the analysis result and at least one of the plurality of sensor image data. In addition, an example of performing overlapping display has been described here, but the display processing unit 18 may also display only the analysis result.

[0082] The analysis unit 17 performs disaster detection processing (step S63). The disaster detection processing is, for example, the above-described processing for extracting a changed area. Here, an example of the analysis performed by the analysis unit 17 is the disaster detection processing, but as described above, the analysis performed by the analysis unit 17 is not limited to the disaster detection processing. The analysis unit 17 stores the analysis result in the processing information storage unit 13.

[0083] In addition, in Figure 1 the example shown, the feature element extraction unit 14 and the feature element extraction units 22 of the first processing unit 15-1 to the nth processing unit 15-n are each separately provided, but they can be regarded as a single feature element extraction unit as a whole. That is, the feature element extraction unit 14 and the feature element extraction units 22 of the first processing unit 15-1 to the nth processing unit 15-n constitute a feature element extraction unit for extracting feature elements that are parts corresponding to reference objects in the target area from map information that is data of a map including the target area and a plurality of observation data. The plurality of observation data are obtained by observing the target area by a plurality of sensors respectively. In addition, in the above example, the feature elements corresponding to each sensor are extracted using simulated feature elements, but any method can be used to extract the feature elements corresponding to a common reference object from the map information and each sensor information, and it is not limited to the above example.

[0084] In addition, the alignment units 23 of the first processing unit 15-1 to the nth processing unit 15-n can be regarded as a single alignment unit as a whole. That is, the alignment units 23 of the first processing unit 15-1 to the nth processing unit 15-n constitute an alignment unit for aligning each of the plurality of observation data so that the position of the feature element corresponding to the observation data coincides with the position of the feature element in the reference data.

[0085] Next, the correspondence processing of the characteristic elements in the corresponding association unit 16 of the present embodiment, that is, the details of the processing in step S61 described above, will be described. Figure 5 It is a flowchart showing an example of the correspondence processing process of the characteristic elements of the present embodiment. As Figure 5 shown, the corresponding association unit 16 sets the variable i representing the number for identifying the sensor to 1 (step S71). Here, the infrared sensor, the visible sensor, the SAR, and the LIDAR are set as the first, second, third, and fourth sensors, respectively.

[0086] The corresponding association unit 16 acquires the evaluation value of each evaluation item of the i-th sensor (step S72). As described above, the characteristics of the sensor vary depending on the type of the sensor, the mounting location of the sensor, etc. In addition, the accuracy of the sensor image data also varies depending on the calibration and correction levels of the sensor image data. Therefore, in the present embodiment, the evaluation value indicating the high or low accuracy of the sensor image data is obtained for each evaluation item.

[0087] As the evaluation items, the following are exemplified.

[0088] (1) Similarity: The similarity between the characteristic elements extracted from each sensor image data and the simulated characteristic elements

[0089] (2) Measurement accuracy: The measurement accuracy in the observation of each sensor

[0090] (3) Position accuracy: The accuracy of the information indicating the position of each sensor

[0091] (4) Pointing accuracy: The accuracy of the information indicating the orientation of each sensor

[0092] (5) Mounting location: The mounting location of each sensor such as fixed on the ground, mounted on a satellite, mounted on a vehicle

[0093] (6) Calibration and correction level: Whether high-precision ground reference objects such as GCPs, natural formations measured by high-precision measuring equipment, etc. are used for high-precision calibration, etc.

[0094] (7) Other conditions such as shooting time period: Whether it is a sensor capable of observing at night, whether it is a sensor capable of observing even in the presence of clouds, etc.

[0095] (2) to (5) and (7) in the above evaluation items basically do not change with each observation and are determined according to the specifications of the sensor and the mounting location, etc. Therefore, for these, the evaluation value is determined in advance for each sensor and stored as characteristic information in the processing information storage unit 13. Generally speaking, according to the height, there is a trade-off relationship between the imaging range and the resolution. Generally speaking, when the specifications of the sensors are of the same level, if the height of the moving body on which the sensor is mounted is high, data with a low resolution is obtained in a large range, and if the height is low, data with a high resolution is obtained in a narrow range. In the case of imaging by a sensor mounted on an aircraft at a height of about 3 to 4.5 km, for example, the imaging range is about 10 km × 10 km. In the case of observation by a sensor mounted on an artificial satellite at a height of 700 km, for example, the observation range is about 70 km × 35 km.

[0096] However, according to the sensor, (2) to (4) may change. Therefore, when (2) to (4) may change, information indicating them may be included in the sensor information, or these information may be acquired separately from the sensor information by the data processing device 1.

[0097] Figure 6 It is a diagram showing an example of the characteristic information of the present embodiment. In Figure 6 the example shown, the characteristic information includes the above (2) measurement accuracy, (3) position accuracy, (4) pointing accuracy, (5) mounting location, and other conditions such as (7) imaging time period (described as "others" in Figure 6 ). In addition, as Figure 6 shown, the characteristic information also includes information related to the map information. This is because, in the present embodiment, in order to compare the accuracy of the map information with that of each sensor information, an evaluation value is also calculated for the map information. As Figure 6 shown, for the map information and each sensor, the evaluation value has been determined in advance for each evaluation item. These information are appropriately updated. For example, (2) to (4) may also be changed after the start of the operation of the sensor or may be different according to the operation mode. Therefore, when they are changed, the evaluation value is set according to the value corresponding to the observation date and time of the sensor information to be processed. In addition, as described above, these information may also be included in the sensor information. When these information are included in the sensor information, the corresponding association unit 16 determines the evaluation value of these evaluation items based on the information included in the sensor information.

[0098] Regarding the (6) correction and calibration levels, it depends on the processing after the observation data is acquired by the sensor. The correction levels for objects with height can also be included in the (6) correction and calibration levels. For example, the observation data of SAR for an object with height contains distortions such as layover and radar shadow. Therefore, the reliability of the observation data varies depending on whether these distortions are corrected. Also, regarding the observation data of sensors other than SAR, even when orthorectification is performed on an object with height, depending on the measurement accuracy of the measured values used in the orthorectification, sometimes, for example, a wall surface with height becomes skewed data. The level corresponding to the correction accuracy of the correction for such an object with height affects the reliability of the data. When the data processing device 1 acquires the sensor information after correction and calibration, information related to the correction and calibration levels is also provided when providing the sensor information. Therefore, the correspondence association unit 16 determines the evaluation value based on this information. When the data processing device 1 performs correction and calibration, each of the first processing units 15-1 to the nth processing unit 15-n notifies the correspondence association unit 16 of the levels of correction and calibration performed in each of the first processing units 15-1 to the nth processing unit 15-n in the data processing device 1, and the correspondence association unit 16 determines the evaluation value based on the notified information. In addition, when the levels of correction and calibration performed by the first processing units 15-1 to the nth processing unit 15-n are predetermined, the evaluation value of the (6) correction and calibration levels can also be included in the characteristic information. Similarly, when the correction and calibration levels of the acquired sensor information are predetermined, the evaluation value of the (6) correction and calibration levels can also be included in the characteristic information.

[0099] In addition, regarding the (1) similarity, the correspondence association unit 16 determines the evaluation value based on the similarities received from each of the first processing units 15-1 to the nth processing unit 15-n. In addition, an example of setting the evaluation value such that the higher the accuracy, the larger the evaluation value is described below. However, the evaluation value can also be set such that the higher the accuracy, the smaller the evaluation value. In the latter case, in the selection of the reference coordinate system based on the comprehensive evaluation value described later, the coordinate system with a lower comprehensive evaluation value is preferentially selected.

[0100] Next, an example of setting the evaluation value will be described. Regarding (1) similarity, the evaluation value is set such that the higher the similarity, the greater the evaluation value. Regarding (2) measurement accuracy, (3) position accuracy, and (4) pointing accuracy, the evaluation value is set such that the higher the accuracy, the greater the evaluation value. Regarding (5) mounting location, the evaluation value is set such that the more stable the location where the sensor is mounted, the greater the evaluation value. For example, the evaluation value is set such that the sensor fixed to the ground has the maximum evaluation value. Hereinafter, the evaluation value is set such that it increases in the order of the sensor mounted on the vehicle, the sensor mounted on the aircraft or satellite, and the sensor mounted on the drone.

[0101] Regarding (6) calibration and calibration level, the evaluation value is set such that the higher the accuracy of the calibration performed, the greater the evaluation value. Regarding (7) other conditions such as the imaging time period, in the case of a visible sensor that cannot observe at night, when the imaging date and time is at night, the comprehensive evaluation value described later is set to 0. In addition, regarding whether it is night, either the data processing device 1 can obtain the sunrise and sunset times, and the corresponding association unit 16 can make a determination based on the imaging date and time and these obtained times, or the sunrise and sunset times can be determined in advance according to the season. Also, in the case of a visible sensor, an infrared sensor, etc. that cannot observe the ground when there are clouds, conditions are determined in advance such that the evaluation value is reduced when it is determined based on other obtained meteorological information, etc. that there are many clouds in the target area. When reducing the evaluation value, the corresponding association unit 16 can either subtract a fixed value from the comprehensive evaluation value described later or multiply the comprehensive evaluation value by a fixed value that is 0 or more and less than 1.

[0102] Return to Figure 5 the description. The corresponding association unit 16 calculates the comprehensive evaluation value of the i-th sensor (step S73). Specifically, the corresponding association unit 16 calculates the comprehensive evaluation value based on the evaluation values of each evaluation item related to the i-th sensor obtained in step S72. For example, the corresponding association unit 16 can either set the sum of the evaluation values of each evaluation item as the comprehensive evaluation value or set the value obtained by adding the evaluation values of each evaluation item with weights as the comprehensive evaluation value. In addition, the weights used in the weighted addition can also include negative values.

[0103] The corresponding association unit 16 determines whether i = n (step S74). When i = n (step S74 "Yes"), the corresponding association unit 16 calculates the comprehensive evaluation value of the map information (step S75). Specifically, the corresponding association unit 16 calculates the comprehensive evaluation value based on the characteristic information, for example.

[0104] As Figure 6As shown, regarding map information, the evaluation values of each evaluation item are also set in advance according to the accuracy of the map information as characteristic information. Regarding the evaluation value of the (2) measurement accuracy of the map information, it is sufficient to use the accuracy of the map information to determine the evaluation value. Regarding the (3) position accuracy, (4) orientation accuracy, and (5) installation location of the map information, there is no need to consider the position and orientation direction as in the case of sensors. Therefore, generally, an evaluation value larger than that of the sensor is set in advance. Regarding the (6) calibration level of the map information, it is also determined in advance according to the type of the map. Generally, the accuracy is higher than that of the sensor information. Therefore, an evaluation value larger than that of the sensor is set in advance. Regarding the (1) similarity of the map information, for example, either the average value of the similarities of each sensor can be used, or the same evaluation value as the highest similarity among the similarities with each sensor can be used, or the evaluation value can be determined in advance.

[0105] When a fixed time has passed after the map information is produced, the possibility of being different from the actual state due to disasters, construction, etc. becomes higher. Therefore, regarding the (7) other conditions such as the shooting time of the map information, it is set in advance in such a way that the longer the elapsed time from the production to the shooting time, the lower the comprehensive evaluation value. For example, the corresponding association unit 16 can either multiply the value obtained by dividing the above-mentioned elapsed time by a fixed value to the comprehensive evaluation value, or multiply the coefficient determined step by step according to the elapsed time to the comprehensive evaluation value. In addition, after a large-scale disaster occurs, after large-scale construction starts, etc., the possibility of the map information being different from the actual situation becomes higher. Therefore, the comprehensive evaluation value of the map information in the corresponding area can also be changed in a decreasing manner. Regarding the change of each evaluation value, it can be carried out either through the input of the user or from an external device via a communication line.

[0106] For example, in the case of using different types of map information by region, etc., the map information to be used is selected according to the target region. In this case, when the accuracy varies according to the type of the map information, the evaluation value of each evaluation item is determined in advance according to the type of the map information.

[0107] Generally, most of the feature elements use map information as reference information for alignment. However, if the alignment of areas with displacements caused by disasters, construction, etc. is based on the map information before these displacements occurred, it is possible to perform image transformation to eliminate the displacements. Therefore, the evaluation items, weights, and evaluation formulas are set in such a way that image transformation to eliminate displacements caused by disasters and construction will not be performed. Regarding the alignment of areas with displacements caused by disasters, construction, etc., for example, the evaluation items, weights, and evaluation formulas are set so that the alignment of each sensor image data is based on the sensor image data with the highest reliability in the sensor that captures images after the displacements caused by disasters and construction occur. At this time, regarding the map information, i.e., the 3D map data, alignment based on the sensor image data with the highest reliability may or may not be performed. When the map information is also aligned based on the sensor data with the highest reliability, the data processing device 1 can achieve the correspondence of the map element name information such as buildings, rail tracks, rivers, etc. associated with the feature elements. Thus, the data processing device 1 can determine what kind of elements the map elements with displacements caused by disasters, construction, etc. are, which can be effectively utilized for analysis.

[0108] Return to Figure 5 In the description of Figure 5 , after step S75, the correspondence association unit 16 selects the coordinate system of the information with the largest comprehensive evaluation value among the map information and the sensor information as the reference coordinate system, and outputs the coordinate values of the feature elements in the reference coordinate system and the coordinate values of the feature elements in the coordinate systems corresponding to the respective sensors (step S76), and ends the feature element correspondence association process. When the result in step S74 is "No", the correspondence association unit 16 increments i by 1 (step S77), and repeats the process starting from step S72.

[0109] In step S76, for example, when the coordinate system of the map information is selected as the reference coordinate system, the coordinate values representing the positions of the feature elements extracted from the map coordinates become the coordinate values of the feature elements in the reference coordinate system. In addition, the correspondence association unit 16 outputs the coordinate values of the feature elements in the coordinate systems of the respective sensors obtained from the first processing unit 15-1 to the nth processing unit 15-n to the respective processing units of the first processing unit 15-1 to the nth processing unit 15-n. On the other hand, for example, as described above, assuming there are 4 sensors, and the coordinate system of the three-dimensional point cloud data of the fourth sensor, i.e., the LIDAR, is selected as the reference coordinate system. In this case, the correspondence association unit 16 sets the coordinate values of the feature elements extracted from the three-dimensional point cloud data obtained from the corresponding fourth processing unit 15-4 as the coordinate values of the feature elements in the reference coordinate system, and outputs them to the first processing unit 15-1 to the fourth processing unit 15-4. Then, the correspondence association unit 16 outputs the coordinate values of the feature elements in the coordinate systems of the respective sensors obtained from the first processing unit 15-1 to the fourth processing unit 15-4 to the respective processing units of the first processing unit 15-1 to the fourth processing unit 15-4.

[0110] When the sensor information is only the information of sensors mounted on artificial satellites, drones, etc. and the map information is new information, generally, the comprehensive evaluation value of the map information is the largest. On the other hand, when the sensor information includes the sensor information of sensors such as LIDAR mounted on vehicles that can be observed with high precision, and in the case where the map information is old or a large-scale disaster has occurred after the map information is made, there may be sensor information with a larger comprehensive evaluation value compared to the comprehensive evaluation of the map information. In the present embodiment, based on the map information and the information with the largest comprehensive evaluation value among the respective sensor information corresponding to the respective sensors, that is, the coordinate system of the information with the largest comprehensive evaluation value is set as the reference coordinate system. Then, the first processing unit 15-1 to the nth processing unit 15-n calculate the local transformation parameters using the coordinate values of the feature elements in the reference coordinate system and the coordinate values of the feature elements in the coordinate systems corresponding to the respective sensors as described above. Then, the first processing unit 15-1 to the nth processing unit 15-n use the local transformation parameters to perform a transformation that matches the sensor image data with the reference coordinate system. Thereby, high-precision alignment of a plurality of sensor image data corresponding to a plurality of sensors can be achieved.

[0111] In addition, when the correspondence association unit 16 selects a coordinate system other than the coordinate system of the map information as the reference coordinate system, the display processing unit 18 may also transform and display the map information in a manner that matches the reference coordinate system when displaying the map information.

[0112] As described above, for each of the map information and the plurality of observation data, the corresponding association unit 16, which is the selection unit of the present embodiment, calculates an evaluation value including the level of accuracy for each evaluation item of the plurality of evaluation items including the similarity, and calculates a comprehensive evaluation value based on the evaluation value of each evaluation item. Then, the corresponding association unit 16 selects reference data based on the comprehensive evaluation values of the map information and the plurality of observation data respectively.

[0113] In addition, regarding the reference object corresponding to the feature element, it is desirable to select an object that is expected to remain unchanged before and after a disaster, etc. However, buildings, roads, railway tracks, rivers, etc. selected as the reference object may change in shape due to collapse, landslide, flooding, etc. In this case, it is difficult to extract the feature element from the sensor image data, and accurate alignment cannot be performed even if forced correspondence is carried out. Therefore, for example, the corresponding association unit 16 calculates the evaluation value after excluding the similarity corresponding to the feature element with low reliability from the evaluation item. When determining whether the reliability of the feature element is low, for example, the similarity to the simulated feature element can be used. The corresponding association unit 16, for example, determines that the reliability of the feature element is low when the similarity is less than the threshold value. Or, for the target area, local transformation parameters obtained during normal times before a disaster are stored in advance. When the reliability of the feature element is low, the corresponding association unit 16 does not perform the corresponding association process and instructs the first processing unit 15-1 to the nth processing unit 15-n to use the normal local transformation parameters. Then, the first processing unit 15-1 to the nth processing unit 15-n perform alignment using the normal local transformation parameters. In addition, when the target area where a disaster is considered to have occurred includes the boundary of the image of the sensor information, alignment is performed on a combined image, which is an image obtained by combining a plurality of adjacent images.

[0114] In addition, for the target area, local transformation parameters obtained during normal times before a disaster can also be stored in advance. When the difference between the normal local transformation parameters and the calculated local transformation parameters for the same feature element is equal to or greater than the threshold value, the alignment unit 23 of the first processing unit 15-1 to the nth processing unit 15-n determines that the feature element has changed due to a disaster or the like. Since the local transformation parameter is each element of the transformation matrix, that is, a plurality of coefficients, the difference of the local transformation parameters is, for example, the sum of the differences of these plurality of coefficients, the weighted average of the differences of each, etc. In this case, the alignment unit 23 of the first processing unit 15-1 to the nth processing unit 15-n performs alignment using the normal local transformation parameters.

[0115] Next, a display example in the display processing unit 18 that uses the result obtained by the alignment of the present embodiment will be described. First, an example of overlapping the analysis result of the analysis unit 17 of the present embodiment on the display screen of the map information will be described. Figure 7This is a diagram showing an example of overlapping the change region extracted by the analysis unit 17 of the present embodiment on the display screen of the map information. In Figure 7 In the example shown, the analysis unit 17 performs a process of extracting a change region that has changed before and after a disaster as the analysis. In Figure 7 In Figure 7 , a change region image 202 indicating a change region extracted using the sensor image data after alignment is superimposed and displayed on the map information image 201 representing the map information. In addition, in the present embodiment, since the sensor image data that has been accurately aligned is used to calculate the change region, it is possible to suppress the positional deviation between the change region and the map information when they are overlapped. As a result, it is possible to prevent the user of the sensor image data from misidentifying the position of the change region.

[0116] Figure 8 This is a diagram showing an example of overlapping the displacement amount calculated by the analysis unit 17 of the present embodiment on the display screen of the map information. In Figure 8 In the example shown, similar to the example shown in Figure 7 the analysis unit 17 performs a process of extracting a change region that has changed before and after a disaster as the analysis. In Figure 8 In the example shown, the analysis unit 17 also calculates the displacement amount of each part. In Figure 8 In Figure 8 , a first region image 204, a second region image 205, and a third region image 206 indicating the displacement amount extracted using the sensor image data after alignment are superimposed and displayed on the map information image 203 representing the map information. In Figure 8 In the example shown, the displacement amount is represented by the density of the hatching. The third region image 206, which represents the region with the largest displacement amount, has the highest hatching density. The second region image 205 represents the region with the next largest displacement amount, and the first region image 204 represents the region with the next largest displacement amount. In Figure 8 In the example shown, as the density of the hatching decreases, that is, as the hatching becomes lighter, the displacement amount decreases. In addition, in Figure 8 although the displacement amount is represented by the density of the hatching in Figure 8 , it is not limited thereto, and the displacement amount may also be represented by color differentiation or the like. In addition, the displacement amount may also be represented by contour lines or the like.

[0117] In addition, when the display processing unit 18 overlays and displays information such as a changed area and a displacement amount on the map information, it can either mechanically overlay and display this information on fixed map data generated for display, or display the map data and this information on different layers. Additionally, the data processing device 1, for example, can either generate map data reflecting the displacement caused by a disaster and overlay and display it on the above-mentioned fixed map data, or display them on different layers. The method of overlay display is not limited to these, and any display suitable for easily communicating the occurrence area of a disaster, etc. may be performed.

[0118] Next, an example of overlay display of sensor image data and analysis results is shown. Figure 9 FIG. is an example of a display screen in which the inundation area calculated by the analysis unit 17 of the present embodiment is overlaid on the visible sensor image data. In Figure 9 the example shown, the analysis unit 17 performs processing to extract the inundation area as an analysis. In Figure 9 FIG., an inundation area image 208 indicating the inundation area extracted using the sensor image data after alignment is overlaid on the sensor image 207 representing the visible sensor image data.

[0119] Although not shown in the figure, similarly, a change area image indicating the change area extracted using the sensor image data after alignment can be overlaid on the sensor image representing the visible sensor image data. The change area image is generated as an image having a shape that is easily visually recognizable, coloring that is easily visually recognizable, etc. Thereby, the visibility of the change area can be improved.

[0120] Not limited to the display examples shown above, the display processing unit 18 can also perform overlay display of any combination. The display processing unit 18 can either overlay and display a plurality of sensor image data corresponding to a plurality of sensors, or overlay and display them with map information. Additionally, at least one of the sensor image data and the map information can be overlaid and displayed with the analysis result, and the analysis result to be overlaid and displayed is not limited to the above example.

[0121] In addition, although an example in which the data processing device 1 includes the analysis unit 17 and the display processing unit 18 has been described above, it can also be configured such that the data processing device 1 does not include the analysis unit 17 and the display processing unit 18, and the processing of the analysis unit 17 and the display processing unit 18 is performed by other devices. In this case, the data processing device 1 sends the map matching information of the sensor image data after alignment to other devices. Alternatively, the data processing device 1 can also record the map matching information on a recording medium and provide the map matching information to other devices through the recording medium.

[0122] As described above, in the present embodiment, the coordinate system of the map information and the information assumed to be highly accurate among the plurality of sensors is selected as the reference coordinate system, and the reference coordinate system is used to align the image data of the plurality of sensors. Therefore, it is possible to accurately align the observation positions corresponding to the data obtained by the plurality of sensors.

[0123] Embodiment 2.

[0124] Figure 10 FIG. is a diagram showing a structural example of the data processing apparatus according to Embodiment 2 of the present invention. The data processing apparatus 1a of the present embodiment includes a correspondence association unit 16a instead of the correspondence association unit 16 of Embodiment 1. The processing information storage unit 13 also stores local transformation parameters. Except for this, it is the same as the data processing apparatus 1 of Embodiment 1. For structural elements having the same functions as those of Embodiment 1, the same reference numerals as those of Embodiment 1 are added to omit the description repeated with Embodiment 1.

[0125] Regarding the reference object observed by the same sensor in the same operation mode and the same pointing direction, if no disaster or the like occurs, almost the same local transformation parameters should be calculated. In the present embodiment, the alignment units 23 of the first processing unit 15-1 to the nth processing unit 15-n each previously store the calculated local transformation parameters in the processing information storage unit 13 according to the sensor, the operation mode, and the pointing direction. Then, the correspondence association unit 16a determines whether the reference object corresponding to the feature element has been displaced based on the difference in the local transformation parameters calculated from the sensor information observed under the same conditions. Since the local transformation parameters are the elements of the transformation matrix, that is, a plurality of coefficients, the difference in the local transformation parameters is, for example, the sum of the differences of these plurality of coefficients, the weighted average of the respective differences, and the like.

[0126] Figure 11 FIG. is a diagram showing an example of the flow of the overall processing in the data processing apparatus 1a of the present embodiment. Steps S11, S12, S21 to S23, S31 to S33, S41 to S43, and S51 to S53 are the same as those in Embodiment 1.

[0127] After steps S23, S33, S43, and S55, the corresponding association unit 16a sets the coordinate system of the map information as the reference coordinate system, and outputs the coordinate values of the feature elements in the reference coordinate system and the coordinate values of the feature elements in the coordinate systems corresponding to the respective sensors (step S61a). That is, the corresponding association unit 16a selects the map information as temporary reference data, and outputs the coordinate values of the feature elements in the reference coordinate system and the coordinate values of the feature elements in the coordinate systems corresponding to the respective sensors to the first processing unit 15-1 to the fourth processing unit 15-4. In the present embodiment, the process of selecting this temporary reference coordinate system and the selection of the reference coordinate system by using the difference from the normal with respect to the local transformation parameters described later are both the feature element corresponding association processes (step S61a).

[0128] The alignment units 23 of the first processing unit 15-1 to the nth processing unit 15-n each calculate local transformation parameters (steps S24, S34, S44, S54) in the same manner as in Embodiment 1, and output the calculated local transformation parameters to the corresponding association unit 16a. That is, the alignment unit 23 calculates the local transformation parameters used in the transformation that makes the positions of the feature elements corresponding to the observation data coincide with the positions of the feature elements in the map information, and outputs the local transformation parameters to the corresponding association unit 16a.

[0129] The corresponding association unit 16a selects the reference coordinate system by using the difference of the local transformation parameters from the normal in addition to the evaluation items described in Embodiment 1 (step S61a). Then, the difference between the first local transformation parameter as the calculated local transformation parameter and the second local transformation parameter as the stored normal local transformation parameter is output to the corresponding association unit 16a.

[0130] Figure 12 is a flowchart showing an example of the feature element corresponding association process of the corresponding association unit 16a of the present embodiment. As Figure 12 shown, the corresponding association unit 16a as the selection unit of the present embodiment sets the coordinate system of the map information as the reference coordinate system, and outputs the coordinate values of the feature elements in the reference coordinate system and the coordinate values of the feature elements in the coordinate systems corresponding to the respective sensors (step S81). Thus, as described above, the local transformation parameters calculated by the respective processing units of the first processing unit 15-1 to the nth processing unit 15-n are received from the respective processing units of the first processing unit 15-1 to the nth processing unit 15-n.

[0131] The corresponding association unit 16a sets i = 1 (step S82), and calculates the difference between the local transformation parameter received from the i-th processing unit and the normal local transformation parameter (step S83). The corresponding association unit 16a determines whether i = n (step S84). If i = n (step S84 "Yes"), it determines whether the difference index is equal to or greater than the threshold value (step S86). Regarding the difference index, for example, the average value, sum, weighted average, maximum value, etc. of n differences corresponding to n sensors can be used. If i ≠ n (step S84 "No"), the corresponding association unit 16a increments i by 1 (step S85), and repeats the processing starting from step S83.

[0132] If the difference index is less than the threshold value (step S86 "No"), the corresponding association unit 16a instructs the first processing unit 15-1 to the n-th processing unit 15-n to directly use the calculated local transformation parameter (step S87), and ends the feature element corresponding association process.

[0133] If the difference index is equal to or greater than the threshold value (step S86 "Yes"), the processes of steps S71 and S72 of Embodiment 1 are performed. After step S72, the corresponding association unit 16a calculates a comprehensive evaluation value (step S88). In the present embodiment, the corresponding association unit 16a calculates the comprehensive evaluation value based on the evaluation value of each evaluation item described in Embodiment 1 and the difference in the local transformation parameter. That is, in the present embodiment, the difference in the local transformation parameter is also included as one of the evaluation items. The difference in the local transformation parameter is the difference between the local transformation parameter received from the i-th processing unit calculated in step S83 and the normal local transformation parameter. Regarding the evaluation value of the difference in the local transformation parameter, it is calculated such that the greater the difference, the greater the evaluation value. Steps S74 and S77 are the same as those in Embodiment 1.

[0134] When the result in step S74 is "Yes", the corresponding association unit 16a selects the coordinate system of the information with the maximum comprehensive evaluation value in the sensor information as the reference coordinate system, and outputs the coordinate values of the feature elements in the reference coordinate system and the coordinate values of the feature elements in the coordinate systems corresponding to the respective sensors (step S89), and ends the feature element corresponding association process. When the local transformation parameters change significantly more than usual, it is highly likely that the reference object corresponding to the feature element has been displaced due to a disaster. Therefore, it can be estimated that the sensor information corresponding to the sensor that detects this displacement can represent the actual state better than the map information. Thus, the coordinate system of the sensor information corresponding to the sensor that detects the displacement is used as the reference coordinate system. In addition, there is also a possibility that the main reason for the significant change in the local transformation parameters compared to usual is a temporary abnormality of the sensor. Therefore, when it can be determined based on other information that the possibility of the sensor being abnormal is high, the evaluation value of the difference in the local transformation parameters of the sensor can also be set to a negative value, etc. For example, when obtaining information indicating a change in the attitude of an artificial satellite carrying the sensor as the sensor information, the evaluation value of the difference in the local transformation parameters can also be set to a negative value when the change exceeds a threshold value.

[0135] In addition, since the map information of the underlying layer is fixed, the deviation between the map layer and the layer of sensor observation data can be used to represent the variation caused by the disaster. The data processing device 1 can also acquire disaster information indicating the date and time and approximate location of the occurrence of the disaster, and based on the disaster information, for the area where the disaster is assumed to have occurred according to the disaster information, the map information created at a date and time before the occurrence of the disaster is not used as the reference information. In this case, the map information can also be used as the reference for areas where the disaster has not occurred. As described in Embodiment 1, when a large-scale disaster occurs, it is also possible to respond by reducing the evaluation value of the map information created before the large-scale disaster, but it is also possible to clearly determine whether it can be used as the reference information as described above.

[0136] Generally, multiple feature elements in the target area are used for alignment, and interpolation processing and the like are performed on parts other than the feature elements as described in Embodiment 1. For example, a situation is also considered where one of the multiple feature elements has shifted from the map information due to a disaster or the like, while the positions of the other feature elements are consistent with the map information. In such a case, if the sensor information of SAR, visible sensor, infrared sensor, and LIDAR observed after the disaster is available, the sensor information with high reliability, that is, high evaluation value, is selected from these sensors as the reference information. In addition, when the target area where the disaster is considered to have occurred includes the boundary of the image of the sensor information, alignment is performed on a combined image obtained by combining multiple adjacent images.

[0137] As described above, for each sensor in the corresponding association unit 16a, alignment is performed using the calculated local transformation parameters, and the difference between the first local transformation parameter, which is the calculated local transformation parameter, and the second local transformation parameter, which is the normally stored local transformation parameter, is calculated. Then, when the difference index calculated based on the difference for each sensor is equal to or greater than the threshold value, an evaluation value including the level of accuracy is calculated for each evaluation item among a plurality of evaluation items including the difference and similarity, and a comprehensive evaluation value is calculated based on the evaluation values for each evaluation item. Further, the corresponding association unit 16a selects reference data based on the comprehensive evaluation values of the map information and each of the plurality of observation data. When the difference index is less than the threshold value, the corresponding association unit 16a selects the map information as the reference data.

[0138] Return to Figure 11 the description of. When each alignment unit 23 of the first processing unit 15-1 to the nth processing unit 15-n receives the information output in the above step S89, the local transformation parameters are recalculated based on the received information (steps S24, S34, S44, S54). Then, each alignment unit 23 of the first processing unit 15-1 to the nth processing unit 15-n performs alignment image transformation processing using the newly calculated local transformation parameters (steps S25, S35, S45, S55). In addition, when each alignment unit 23 of the first processing unit 15-1 to the nth processing unit 15-n receives the instruction in step S87, alignment image transformation processing is performed using the already calculated local transformation parameters (steps S25, S35, S45, S55).

[0139] That is, when the map information is selected as the reference data, the alignment unit 23 performs alignment using the first local transformation parameter. In addition, when data other than the map information is selected as the reference data, a third local transformation parameter used in the transformation for making the position of the feature element corresponding to the observation data coincide with the position of the feature element in the reference data is calculated, and alignment is performed using the third local transformation parameter.

[0140] The processing in steps S25, S35, S45, and S55 is the same as that in Embodiment 1. The processing after step S61a performed again based on the received information is the same as that in Embodiment 1. Steps S26, S36, S46, S56, S62, S63, and S64 are also the same as those in Embodiment 1.

[0141] As described above, in the present embodiment, local transformation parameters are calculated based on map information, and when the difference index calculated based on the difference between the calculated local transformation parameters and the normal local transformation parameters is equal to or greater than a threshold value, a reference coordinate system is reselected. Therefore, even when the reference object corresponding to the feature element is displaced due to a disaster or the like, the alignment of the sensor image data can be performed with high accuracy.

[0142] Embodiment 3.

[0143] Figure 13 FIG. is a diagram showing a structural example of a data processing apparatus according to Embodiment 3 of the present invention. The data processing apparatus 1b of the present embodiment includes a selection unit 19, and includes a feature element extraction unit 14a and a correspondence association unit 16b in place of the feature element extraction unit 14 and the correspondence association unit 16 of Embodiment 1. Other than that, it is the same as the data processing apparatus 1 of Embodiment 1. For structural elements having the same functions as those of Embodiment 1, the same reference numerals as those of Embodiment 1 are added to omit the description repeated with Embodiment 1.

[0144] In Embodiment 1, the simulated feature elements were generated based on the feature elements extracted from the map information. However, in the present embodiment, the simulated feature elements are generated based on the feature elements extracted by the sensor assumed to have the highest accuracy among the multiple sensors. Then, the feature element extraction units 22 of the first processing unit 15-1 to the nth processing unit 15-n extract the feature elements based on the simulated feature elements.

[0145] Generally, the sensors mounted on a vehicle can observe the reference object from a short distance, have a stable posture, and high position accuracy. Therefore, compared with the sensors mounted on drones, artificial satellites, etc., high-accuracy observation data can be obtained. In the case where there is sensor information corresponding to such a sensor assumed to have high accuracy, in the present embodiment, the simulated feature elements are generated based on the feature elements extracted by the sensor assumed to have the highest accuracy.

[0146] An example of the sensor assumed to have the highest accuracy is a LIDAR mounted on a vehicle to obtain three-dimensional point cloud data, but it is not limited thereto. Hereinafter, Figure 3 As in the example shown, it is assumed that there are 4 sensors: SAR, visual sensor, infrared sensor, and LIDAR, and the case where the sensor assumed to have the highest accuracy is LIDAR will be described as an example.

[0147] Figure 14It is a flowchart showing an example of the simulated feature element generation process of this embodiment. The selection unit 19 determines whether there is LIDAR sensor information corresponding to the area and time period to be processed (step S91). When there is LIDAR sensor information corresponding to the area and time period to be processed (step S91 "Yes"), the selection unit 19 selects the coordinate system corresponding to the LIDAR as the reference coordinate system (step S92). That is, when there is observation data of the sensor with the highest predetermined accuracy among multiple sensors, the selection unit 19 selects this observation data as the reference data. The selection unit 19 notifies the feature element extraction unit 14a that the coordinate system corresponding to the LIDAR is set as the reference coordinate system.

[0148] The feature element extraction unit 14a extracts feature elements from the map information in the same manner as the feature element extraction unit 14 of the first embodiment. The feature element extraction unit 14a generates simulated feature elements corresponding to the LIDAR based on the feature elements extracted from the map information (step S93). The feature element extraction unit 14a outputs the generated simulated feature elements to the fourth processing unit 15-4 corresponding to the LIDAR. The fourth processing unit 15-4 extracts feature elements from the three-dimensional point cloud data corresponding to the LIDAR using the simulated feature elements in the same manner as the first embodiment (step S94). The fourth processing unit 15-4 outputs the extracted feature elements to the feature element extraction unit 14a. The feature element extraction unit 14a generates simulated feature elements corresponding to each sensor based on the feature elements extracted by the fourth processing unit 15-4 (step S95), and ends the simulated feature element generation process.

[0149] When there is no LIDAR sensor information corresponding to the area and time period to be processed (step S91 "No"), the coordinate system corresponding to the map information is selected as the reference coordinate system (step S96). The feature element extraction unit 14a generates simulated feature elements corresponding to each sensor based on the feature elements extracted from the map information in the same manner as the first embodiment (step S97), and ends the simulated feature element generation process. After the simulated feature element generation process, the feature element extraction unit 14a outputs the generated simulated feature elements to the first processing unit 15-1 to the fourth processing unit 15-4 respectively.

[0150] Figure 15 It is a diagram showing an example of the overall processing flow in the data processing device 1b of this embodiment. Figure 15An example is shown in which it is determined in step S91 described above that there is sensor information of the LIDAR corresponding to the area and time period to be processed. Steps S11, S21 to S22, S31 to S32, S41 to S42, and S51 to S52 are the same as those in Embodiment 1. The feature element extraction unit 14a generates a simulated feature element corresponding to the LIDAR based on the feature elements selected from the map information and outputs it to the fourth processing unit 15-4 (step S12a). Then, the fourth processing unit 15-4 uses the simulated feature element to extract feature elements from the three-dimensional point cloud data corresponding to the LIDAR in the same manner as in Embodiment 1 (step S53).

[0151] The feature element extraction unit 14a uses the feature elements extracted by the fourth processing unit 15-4 to generate simulated feature elements corresponding to the respective sensors and outputs them to the first processing unit 15-1 to the fourth processing unit 15-4 respectively (step S65). Steps S23, S33, and S43 are the same as those in Embodiment 1. As the feature element correspondence association process, the correspondence association unit 16b sets the coordinate system of the LIDAR as the reference coordinate system and outputs the coordinate values of the feature elements in the reference coordinate system and the coordinate values of the feature elements in the coordinate systems corresponding to the respective sensors (step S61b). The processes of steps S24 to S26, S34 to S36, S44 to S46, S54 to S56, and S62 to S64 are the same as those in Embodiment 1.

[0152] In Figure 15 In the case where it is determined in step S91 described above that there is no sensor information of the LIDAR corresponding to the area and time period to be processed, the overall process except for the feature element correspondence association process in step S61 of Embodiment 1 is the same as the process shown in Figure 3 In the case where it is determined that there is no sensor information of the LIDAR corresponding to the area and time period to be processed, in the present embodiment, the correspondence association unit 16b sets the coordinate system of the map coordinates as the reference coordinate system and outputs the coordinate values of the feature elements in the reference coordinate system and the coordinate values of the feature elements in the coordinate systems corresponding to the respective sensors, instead of step S61 of Embodiment 1.

[0153] As described above, in the present embodiment, in the case where there is sensor information of the sensor assumed to have the highest accuracy among the multiple sensors, alignment is performed based on the sensor assumed to have the highest accuracy. Therefore, the alignment of the sensor image data can be performed with high accuracy.

[0154] The structure shown in the above embodiment represents an example of the content of the present invention, and it can be combined with other known technologies, and a part of the structure can also be omitted or changed without departing from the gist of the present invention.

Claims

1. A data processing device, characterized in that, Comprising: A feature element extraction unit that extracts, from map information which is data of a map including a target area and multiple observation data obtained by respectively observing the target area by multiple sensors, a feature element which is a part corresponding to a reference object in the target area; A selection unit that selects one of the map information and the multiple observation data as reference data to be set as an alignment reference; And An alignment unit that aligns each of the multiple observation data in such a way that the position of the feature element corresponding to the observation data coincides with the position of the feature element in the reference data; The feature element extraction unit uses the feature element extracted from the map information to generate, for each sensor, a simulated feature element that simulates the data obtained by observing the reference object by the sensor, and uses the corresponding simulated feature element for each sensor to extract a feature element from the observation data corresponding to the sensor.

2. The data processing device according to claim 1, wherein The feature element extraction unit calculates, for each sensor, the similarity between the simulated feature element and the feature element extracted from the observation data; The selection unit calculates, for each evaluation item of multiple evaluation items including the similarity, an evaluation value including the level of accuracy for each of the map information and each of the multiple observation data, calculates a comprehensive evaluation value based on the evaluation value of each evaluation item, and selects the reference data based on the comprehensive evaluation value of each of the map information and the multiple observation data.

3. The data processing device according to claim 1, wherein The feature element extraction unit calculates, for each sensor, the similarity between the simulated feature element and the feature element extracted from the observation data; The alignment unit calculates, for each sensor, local transformation parameters used in a transformation for making the position of the feature element corresponding to the observation data coincide with the position of the feature element in the map information; The selection unit performs the alignment for each sensor using the calculated local transformation parameters, calculates the difference between a first local transformation parameter which is the calculated local transformation parameter and a second local transformation parameter which is the usual local transformation parameter stored; When the difference index calculated based on the difference for each sensor is equal to or greater than a threshold, the selection unit calculates, for each evaluation item of multiple evaluation items including the difference and the similarity, an evaluation value including the level of accuracy, calculates a comprehensive evaluation value based on the evaluation value of each evaluation item, and selects the reference data based on the comprehensive evaluation value of each of the map information and the multiple observation data; When the difference index is less than the threshold, the selection unit selects the map information as the reference data. When the map information is selected as the reference data, the alignment unit performs the alignment using the first local transformation parameter. When information other than the map information is selected as the reference data, the alignment unit calculates a third local transformation parameter used in a transformation that aligns the position of the feature element corresponding to the observation data with the position of the feature element in the reference data, and performs the alignment using the third local transformation parameter.

4. A data processing device, characterized in that, comprising: a feature element extraction unit that extracts, from map information that is data of a map including an object area and a plurality of observation data obtained by respectively observing the object area with a plurality of sensors, a feature element that is a part corresponding to a reference object in the object area; a selection unit that selects one of the map information and the plurality of observation data as reference data to be set as an alignment reference; and an alignment unit that aligns each of the plurality of observation data so that the position of the feature element corresponding to the observation data coincides with the position of the feature element in the reference data; when there is observation data of a sensor with the highest predetermined accuracy among the plurality of sensors, the selection unit selects the observation data as the reference data; the feature element extraction unit uses the feature element extracted from the reference data to generate, for each sensor, a simulated feature element that simulates data obtained by observing the reference object with the sensor, and extracts a feature element from the observation data corresponding to the sensor using the corresponding simulated feature element.

5. The data processing device according to claim 4, wherein the sensor with the highest accuracy is a lidar mounted on a vehicle to acquire three-dimensional point cloud data.

6. The data processing device according to any one of claims 1 to 5, wherein the reference object is at least one of a plurality of ground reference points, a naturally formed structure, and a building.

7. The data processing device according to claim 6, wherein the reference object is a linear structure.

8. The data processing device according to any one of claims 1 to 5, wherein the data processing device includes an analysis unit that performs analysis to extract a disaster-occurring part using the observation data on which the alignment has been performed.

9. The data processing device according to claim 8, wherein the data processing device includes a display processing unit that overlays and displays the result of the analysis with at least one of the plurality of observation data on which the alignment has been performed and the map information.

10. The data processing device according to claim 9, wherein the display processing unit highlights the result of the analysis.

11. The data processing device according to any one of claims 1 to 5, wherein the data processing device includes a display processing unit that overlays and displays at least one of the plurality of observation data on which the alignment has been performed and the map information.

12. The data processing device according to any one of claims 1 to 5, characterized in that: The plurality of sensors include sensors mounted on artificial satellites or aircraft.

13. The data processing device according to claim 12, characterized in that: The plurality of sensors include synthetic aperture radars.

14. A data processing method, characterized in that, Comprising the following steps: First step, the data processing device extracts, as characteristic elements corresponding to a reference object in the object area, characteristic elements from map information that is data of a map including the object area and from a plurality of observation data obtained by respectively observing the object area by the plurality of sensors; Second step, the data processing device selects one of the map information and the plurality of observation data as reference data to be used as a reference for alignment; And Third step, for each of the plurality of observation data, the data processing device aligns the observation data in such a way that the position of the characteristic element corresponding to the observation data coincides with the position of the characteristic element in the reference data; In the first step, the data processing device uses the characteristic elements extracted from the map information to generate, for each sensor, simulated characteristic elements that simulate the data obtained by observing the reference object by the sensor, and uses the corresponding simulated characteristic elements for each sensor to extract characteristic elements from the observation data corresponding to the sensor.

Images