Four-dimensional mapping information generation system

Abstract

Using three-dimensional underground information as a benchmark, we accurately identify and extract three-dimensional positional information above ground. [Solution] The system includes a subterranean exploration means 20 that generates subterranean three-dimensional information beneath the road surface on which the exploration vehicle 10 travels, a ground image generation means 30 that generates ground-level three-dimensional images on the road surface on which the exploration vehicle 10 travels, a unification processing unit 50 that generates unified information from the subterranean to the ground space by a predetermined data unification processing, an unification information storage unit 61 that stores multiple unified pieces of information generated at different times, and an unification information extraction unit 62 that extracts multiple unified pieces of information from the stored unified pieces of information that are generated at different times and whose ground positions coincide based on the subterranean three-dimensional information, thereby generating four-dimensional mapping information by adding time axis information to the unified three-dimensional information from the subterranean and ground.

Description

Technical Field

[0001] The present invention relates to a four-dimensional mapping information generation system that can generate information by unifying three-dimensional information in the ground generated by underground exploration technology and three-dimensional images on the ground acquired by an in-vehicle camera or the like, and further generate four-dimensional mapping information with time-axis information added to the unified information of the ground and the ground.

Background Art

[0002] In recent years, natural disasters such as earthquakes, typhoons, concentrated heavy rains, and local heavy rains have occurred frequently, and it is expected that they will become even more severe in the future and large-scale disasters will occur. Measures against large-scale disasters, especially efforts for pre-disaster recovery, have become an urgent issue and an immediate necessity. Pre-disaster recovery is an effort to take measures to minimize damage in the event of damage caused by an earthquake. As early recovery and reconstruction measures for large-scale earthquakes such as the Nankai Trough earthquake and the earthquake directly beneath the capital that are expected to occur in the future, in addition to disaster prevention and mitigation measures, the effort of pre-disaster recovery to make preparations assuming post-disaster recovery in advance is becoming increasingly important.

[0003] Here, as an issue regarding the recovery and reconstruction of the disaster-stricken area, the accurate position of buried objects in the ground such as water pipes may not be grasped. For example, in the Noto Peninsula earthquake that occurred in Reiwa 6, significant damage occurred to roads and water supply, and the water supply was cut off in multiple municipalities for several months, which is still fresh in memory as a major obstacle to the reconstruction of life. As factors contributing to the time-consuming restoration of the water supply, in addition to the fact that the investigation of the aging of water pipes buried in the ground has not been sufficiently carried out during normal times, it is also difficult to accurately grasp the positions of various buried objects including water pipes intricately laid in the ground even after the disaster. And in the disaster-stricken area, due to the delay in water supply restoration, even if houses and the like are intact, many disaster victims have no choice but to continue taking refuge in evacuation areas and shelters, and the delay in early recovery has become a major problem in terms of the outflow of the local population.

[0004] Furthermore, during the reconstruction efforts following the Great East Japan Earthquake, the fact that the reference points used as the basis for surveying shifted by several meters due to the earthquake became a major obstacle to the progress of reconstruction plans, particularly the reconstruction of homes that had been swept away by the tsunami. For housing reconstruction, it is first necessary to determine the boundaries of each property, but if all above-ground structures are swept away by a tsunami, not only are there no reference points, but there are also no buildings or structures that can be used as references. In that case, even if one tries to adjust based on coordinates, if the reference points are shifted or have disappeared, the coordinate (absolute coordinate) information will also be useless. As a result, until correct reference points become available, surveying to determine the boundaries cannot begin for, say, several months, and housing reconstruction will be greatly delayed.

[0005] Therefore, in Japan, a country prone to earthquakes, including the Nankai Trough earthquake that is expected to occur in the future, achieving early recovery and reconstruction after a disaster requires measures and surveying techniques that do not rely on surveying reference points, such as the ability to quickly and accurately grasp underground information linked to surface information, and to quickly determine the accurate position on the surface even if surveying reference points are shifted or lost due to earthquakes or tsunamis.

[0006] Furthermore, infrastructure such as roads, bridges, tunnels, and sewage systems, which were developed during the period of rapid economic growth, will see an accelerating increase in the proportion of facilities that will deteriorate in the future. Such aging infrastructure can lead to serious problems such as accidents that endanger human lives and disruptions to lifelines, making it an urgent issue to address aging infrastructure. Furthermore, work is underway to replace aging buried pipes such as water and sewage lines, and to remove utility poles, particularly along transportation routes that are crucial for saving lives and transporting relief supplies during disasters, which have become increasingly severe and frequent in recent years. Managing and maintaining such infrastructure requires considerable expense, time, and effort, and effective responses and measures are needed to efficiently carry out construction and other related work.

[0007] In underground exploration technology, which involves searching for buried objects and other structures beneath the ground, in order to reduce the burden of location identification work—which involves determining the position of underground objects such as cavities and buried pipes shown in the underground exploration data relative to the ground surface—the underground exploration data is displayed together with maps or road surface images showing the corresponding ground surface, thereby identifying the location and size of cavities, buried pipes, etc., as their positions on the map or road surface image. Examples of technologies that display such ground-penetrating exploration data together with maps or road surface images showing the corresponding ground surface include the "ground-penetrating exploration method" disclosed in Patent Document 1. [Prior art documents] [Patent Documents]

[0008] [Patent Document 1] Japanese Patent Application Publication No. 11-352223 [Non-patent literature]

[0009] [Non-Patent Document 1] David J. Daniels, Ground Penetrating Radar, The Institution of Engineering and Technology, 2004 [Non-Patent Document 2] Richard Hartley, Andrew Zisserman, Multiple view geometry in computer vision, Cambridge University Press, 2003 [Non-Patent Document 3] P. Alcantarilla, J. Nuevo, and A. Bartoli, Fast explicit diffusion for accelerated features in nonlinear scale spaces, British Machine Vision Conference, pp.13.1-13.11, 2013 [Non-Patent Document 4] Lindenberger, Philipp, Paul-Edouard Sarlin, and Marc Pollefeys, Lightglue: Local feature matching at light speed, Proceedings of the IEEE / CVF International Conference on Computer Vision, 2023 [Non-Patent Document 5] Manabu Hashimoto, Shuichi Akizuki, and Shoichi Takei, Research Trends in Three-Dimensional Features for Object Recognition, Transactions of the Institute of Electrical Engineers of Japan (Electronics, Information and Systems Division), Vol. 136, No. 8, pp. 1038-1046, 2016. [Non-Patent Document 6] Yang, J., Zhang, J., Cai, Z. et al., Novel 3D local feature descriptor of point clouds based on spatial voxel homogenization for feature matching, Vis. Comput. Ind. Biomed. Art 6, 18, 2023 [Overview of the Initiative] [Problems that the invention aims to solve]

[0010] However, conventional ground-penetrating exploration technologies, including the technology disclosed in Patent Document 1, only display ground-penetrating exploration data obtained from ground-penetrating radar, etc., alongside corresponding maps or road surface images. They were unable to accurately link and identify the location of the explored cavities, buried pipes, etc., on the ground surface. Therefore, it is difficult to accurately determine which building or road on the surface corresponds to which location in the ground-penetrating survey data. Furthermore, if the ground-penetrating survey data is destroyed, swept away, or lost due to the disaster, it becomes difficult not only to associate the ground-penetrating survey data with the ground-penetrating location information, but also to accurately identify and grasp the ground-penetrating location information itself.

[0011] Thus, with conventional ground-penetrating exploration technology, it has not been easy to accurately correlate and grasp location information above and below ground. In particular, if surveying reference points have moved or disappeared, it becomes impossible to determine location information unless the reference points are re-established, making rapid recovery and reconstruction after a disaster extremely difficult. Furthermore, from the perspective of pre-disaster reconstruction, there is a strong need to improve the efficiency and reduce costs of work related to the management and replacement of buried pipes such as water and sewage systems, which will deteriorate rapidly in the future. However, it has been difficult to take effective measures with current ground-penetrating exploration technologies.

[0012] As a result of diligent research, the inventors of this invention have come up with the present invention, which integrates three-dimensional information from underground and three-dimensional information from above ground, and further adds time-axis information to it. By utilizing the three-dimensional information from underground as benchmark and mapping information, the invention can effectively solve the problems of responding to the recovery and reconstruction of disaster-stricken areas and managing aging infrastructure.

[0013] In other words, the present invention was proposed to solve the problems of the conventional technology described above, and aims to provide a four-dimensional mapping information generation system that generates unified three-dimensional information by combining three-dimensional information of the ground generated by underground exploration technology and three-dimensional images of the ground acquired by an in-vehicle camera, etc., and further generates four-dimensional information by adding time axis information to the unified three-dimensional information of the ground and ground, thereby enabling accurate identification and extraction of three-dimensional positional information of the ground using the three-dimensional information of the ground as a benchmark and mapping information, and enabling comparison of three-dimensional information of the ground in peacetime and after a disaster, for example, thereby improving the efficiency of infrastructure management in peacetime and realizing early recovery and reconstruction of disaster-stricken areas, and in particular provides effective and useful information for pre-disaster reconstruction and the management of aging infrastructure. [Means for solving the problem]

[0014] To achieve the above object, the four-dimensional mapping information generation system of the present invention includes a subsurface exploration means disposed on an exploration vehicle capable of traveling on a road surface, for generating three-dimensional information of the subsurface under the road surface on which the exploration vehicle travels; a ground image generation means disposed on the exploration vehicle, for generating a three-dimensional ground image on the road surface on which the exploration vehicle travels; a unification processing means for unifying the three-dimensional subsurface information generated by the subsurface exploration means and the three-dimensional ground image generated by the ground image generation means into unified information from the subsurface to the ground space by a predetermined data unification process; a unified information storage means for attaching and storing time-axis information indicating the generation time to a plurality of the unified information with different generation times; and a unified information extraction means for extracting a plurality of unified information with different generation times and the same ground position based on the three-dimensional subsurface information from the stored plurality of the unified information. The system is configured to generate four-dimensional mapping information with time-axis information attached to the unified three-dimensional information of the subsurface and the ground.

Advantages of the Invention

[0015] According to the present invention, it is possible to generate information by unifying the three-dimensional subsurface information generated by the subsurface exploration technology and the three-dimensional ground image acquired by an in-vehicle camera or the like, and further generate four-dimensional information with time-axis information attached to the three-dimensional unified information of the subsurface and the ground. Thereby, by accurately specifying and extracting the three-dimensional ground position information using the three-dimensional subsurface information as benchmark mapping information, for example, by generating and outputting the three-dimensional ground information before and after a disaster so that they can be compared, it becomes possible to accurately specify and grasp the three-dimensional ground information before the disaster. Therefore, according to the present invention, it is possible to improve the efficiency of infrastructure management in normal times and realize the early recovery and reconstruction of the disaster-stricken area during a disaster. In particular, it is possible to provide effective and useful information for the management of pre-disaster reconstruction and aging infrastructure.

Brief Description of the Drawings

[0016] [Figure 1]It is an explanatory diagram schematically showing an overall picture of underground exploration by an underground exploration device in a four-dimensional mapping information generation system according to an embodiment of the present invention. [Figure 2] It is a perspective view showing a part of the underground under the road surface on which a exploration vehicle travels, schematically showing the exploration vehicle that acquires and generates underground exploration data of an underground exploration device in a four-dimensional mapping information generation system according to an embodiment of the present invention. [Figure 3] It is an explanatory diagram schematically showing an operation image of an exploration vehicle that acquires and generates underground exploration data of an underground exploration device in a four-dimensional mapping information generation system according to an embodiment of the present invention. (a) is a cross-sectional view of the road surface on which the exploration vehicle travels, and (b) is a plan view of the same road surface on which the exploration vehicle travels. [Figure 4] It is an explanatory diagram schematically showing the positional relationship between an exploration vehicle and sensors of an underground exploration device in a four-dimensional mapping information generation system according to an embodiment of the present invention. (a) and (b) are bottom views of the exploration vehicle, and (c) is a plan view of the road surface showing the exploration range of the sensors corresponding to (a) and (b). [Figure 5] It is an explanatory diagram schematically showing underground exploration data acquired and generated by an underground exploration device in a four-dimensional mapping information generation system according to an embodiment of the present invention. (a) is an image diagram of three-dimensional underground data acquired by a sensor, and (b) is an image diagram of an output image generated and output based on the data acquired by the sensor. [Figure 6] It is an explanatory diagram schematically showing an imaging range of a 360-degree full-circle three-dimensional video on the ground acquired by an underground exploration device in a four-dimensional mapping information generation system according to an embodiment of the present invention. (a) shows the imaging range on a road surface, bridge, etc., and (b) shows the imaging range of a slope (normal plane) on the side of the road. Each is a front cross-sectional view of the road surface. Also, (c) is an example of a 360-degree full-circle video. [Figure 7] It is a flowchart showing the procedure for unifying underground exploration data and ground video in an underground exploration device in a four-dimensional mapping information generation system according to an embodiment of the present invention. [Figure 8]This is a block diagram showing the basic configuration of the four-dimensional mapping information generation unit in a four-dimensional mapping information generation system according to one embodiment of the present invention. [Figure 9] This is an illustrative diagram of the subsurface three-dimensional data and the image obtained by unifying the subsurface three-dimensional data into a ground orthomosaic image, which are acquired and generated by a four-dimensional mapping information generation system according to one embodiment of the present invention, where (a) is the data for investigating cavities beneath the road surface and (b) is the data for investigating rebar corrosion in bridge decks. [Figure 10] This is an illustrative diagram of an image obtained by unifying and processing subsurface 3D data, ground orthomosaic images, real ground images, and map images acquired and generated by a four-dimensional mapping information generation system according to one embodiment of the present invention. [Figure 11] This is an illustrative diagram of an image of four-dimensional mapping information acquired and generated by a four-dimensional mapping information generation system according to one embodiment of the present invention, showing a comparison of images of the same location during normal times and after a disaster. [Figure 12] This is a block diagram showing the basic configuration of a camera vector (CV) calculation unit that generates a camera vector image from a 360-degree stereoscopic image of the ground in a ground-penetrating exploration device, in a four-dimensional mapping information generation system according to one embodiment of the present invention. [Figure 13] This is an explanatory diagram showing a transformed image obtained from a 360-degree stereoscopic image of the ground. (a) shows a virtual sphere onto which a spherical image is superimposed, (b) shows an example of a spherical image superimposed on the virtual sphere, and (c) shows the spherical image shown in (b) unfolded into a plane according to the Mercator projection. [Figure 14] This is an explanatory diagram showing a specific method for detecting camera vectors in the CV calculation unit according to one embodiment of the present invention. [Figure 15] This is an explanatory diagram showing a specific method for detecting camera vectors in the CV calculation unit according to one embodiment of the present invention. [Figure 16] This is an explanatory diagram showing a specific method for detecting camera vectors in the CV calculation unit according to one embodiment of the present invention. [Figure 17]This is an explanatory diagram showing a desirable feature point specification method in a camera vector detection method by a CV calculation unit according to one embodiment of the present invention. [Figure 18] This graph shows an example of the three-dimensional coordinates of feature points and camera vectors obtained by the CV calculation unit according to one embodiment of the present invention. [Figure 19] This graph shows an example of the three-dimensional coordinates of feature points and camera vectors obtained by the CV calculation unit according to one embodiment of the present invention. [Figure 20] This graph shows an example of the three-dimensional coordinates of feature points and camera vectors obtained by the CV calculation unit according to one embodiment of the present invention. [Figure 21] This is an explanatory diagram showing a case in which a CV calculation unit according to one embodiment of the present invention sets multiple feature points according to the distance from the camera to the feature points and repeatedly performs multiple calculations. [Figure 22] This figure shows the trajectory of the camera vector, obtained by the CV calculation unit according to one embodiment of the present invention, displayed in a video image. [Figure 23] This is an explanatory diagram illustrating an implementation method for generating four-dimensional mapping information in a four-dimensional mapping information generation system according to one embodiment of the present invention, by inputting two pieces of ground-penetrating survey measurement information, each with positional information for a first coordinate system and a second coordinate system, and three-dimensional ground mapping information. [Figure 24] This image shows a grid-based cross-sectional view and a cross-sectional view based on latitude and longitude, of the horizontal cross-sectional view of the ground-penetrating radar reflection image in a four-dimensional mapping information generation system according to one embodiment of the present invention. [Figure 25] This is an explanatory diagram illustrating an implementation method for a four-dimensional mapping information generation system according to one embodiment of the present invention, which extracts characteristic locations of subsurface three-dimensional information linked to horizontal plane coordinates and transforms local coordinates based on the correspondence with the position of a model used as a reference point. [Figure 26] This figure shows the elements and relationships of a linear model created from a horizontal cross-sectional view of subsurface three-dimensional information linked to a first and second coordinate system, in a four-dimensional mapping information generation system according to one embodiment of the present invention. [Figure 27]This figure shows the elements and relationships of a polyline-shaped model created from a horizontal cross-sectional view of subsurface three-dimensional information linked to a first and second coordinate system, in a four-dimensional mapping information generation system according to one embodiment of the present invention. [Figure 28] This figure shows the elements and relationships of a curved shape model created from a horizontal cross-sectional view of subsurface three-dimensional information linked to a first and second coordinate system, in a four-dimensional mapping information generation system according to one embodiment of the present invention. [Figure 29] This is an explanatory diagram illustrating how subsurface three-dimensional information linked to a coordinate system is divided into multiple regions in a four-dimensional mapping information generation system according to one embodiment of the present invention. [Figure 30] This is an illustrative diagram of an implementation for comparing two sets of three-dimensional underground information within a region linked to a first and second coordinate system, in a four-dimensional mapping information generation system according to one embodiment of the present invention. [Figure 31] This is a schematic cross-sectional view showing the correspondence of depths in two coordinate systems using representative reflection signals in a four-dimensional mapping information generation system according to one embodiment of the present invention. [Figure 32] This is an explanatory diagram illustrating the correspondence between locations with identical local features based on straight lines in horizontal cross-sectional views of two ground-penetrating radars in a four-dimensional mapping information generation system according to one embodiment of the present invention. [Figure 33] This is an explanatory diagram showing the corresponding positions and displacement amounts of two images based on the local features of horizontal cross-sectional views at multiple depths of ground-penetrating radar reflection images in a four-dimensional mapping information generation system according to one embodiment of the present invention. [Figure 34] This figure shows a model of the three-dimensional shape of a cavity extracted from three-dimensional underground information in a four-dimensional mapping information generation system according to one embodiment of the present invention. [Figure 35] This is an explanatory diagram illustrating the situation in a four-dimensional mapping information generation system according to one embodiment of the present invention, where the amount of variation has multiple values ​​at different depths. [Figure 36]This is an explanatory diagram illustrating an implementation method for a four-dimensional mapping information generation system according to one embodiment of the present invention, which generates mapping information of road boundaries and boundary information from drawings based on camera images, transforms the positions of elements in the mapping information, and incorporates them as mapping information in a second coordinate system. [Modes for carrying out the invention]

[0017] Hereinafter, embodiments of the four-dimensional mapping information generation system according to the present invention will be described with reference to the drawings. [Four-dimensional mapping information generation system] Here, the generation, output, and unification processing of subterranean three-dimensional information and terrestrial three-dimensional images, the extraction and generation of four-dimensional mapping information, and camera vector calculations in the four-dimensional mapping information generation system of the present invention described below are realized by processes, means, and functions executed by a computer according to the instructions of a program (software). The program can send instructions to each component of the computer to perform the predetermined processes and functions according to the present invention described below. In other words, each process, means, and function in this invention is realized by specific means involving the cooperation of a program and a computer.

[0018] Furthermore, all or part of the program is provided on, for example, a magnetic disk, optical disk, semiconductor memory, or any other computer-readable recording medium, and the program read from the recording medium is installed on the computer and executed. Furthermore, the program can be loaded and executed directly onto a computer via a communication line without using a recording medium. The ground-penetrating exploration device according to the present invention can also consist of a single information processing device (e.g., a single personal computer) or multiple information processing devices (e.g., a group of computers).

[0019] [Ground-penetrating exploration equipment] Figure 1 shows a schematic configuration of a ground-penetrating exploration device that constitutes a four-dimensional mapping information system according to one embodiment of the present invention. The underground exploration device according to one embodiment of the present invention shown in the figure comprises an exploration vehicle 10, a three-dimensional ground-penetrating radar 20 and an omnidirectional camera 30 mounted on the exploration vehicle 10. As shown in Figure 1, the exploration vehicle 10 travels along any road surface to be explored, acquiring and generating three-dimensional underground information beneath the road surface using a three-dimensional ground-penetrating radar, and acquiring and generating three-dimensional ground images above the road surface using an omnidirectional camera 30. Subsequently, the three-dimensional underground information and the three-dimensional ground images are combined and output as a single, unified three-dimensional information (see Figure 9, described later). The following describes in detail the components of the underground exploration device.

[0020] [Exploration Vehicle 10] The exploration vehicle 10 is a vehicle capable of traveling on roads, ground, or other surfaces that are the target and area of ​​the subsurface exploration, and can consist of vehicles such as passenger cars, trucks, light vans, various work vehicles, and minicars. However, the specific configuration of the exploration vehicle 10 is not particularly limited, as long as it can travel on the desired road surface that is the target and purpose of the exploration. Figures 2 and 3 schematically show the operation of the exploration vehicle 10 in this embodiment. As shown in Figure 3(a), the exploration vehicle 10 has a GPS (high-precision GNSS) 11 mounted on the roof, an antenna 22 for a three-dimensional ground-penetrating radar 20 on the bottom of the vehicle, and an omnidirectional camera 30 positioned at the rear of the roof. Furthermore, the exploration vehicle 10 houses the radar unit 21 of the three-dimensional ground-penetrating radar 20, as well as a CV calculation unit 40 and an integrated processing unit 50.

[0021] Furthermore, although not shown in Figure 3, the exploration vehicle 10 may also be equipped with a four-dimensional mapping information generation unit 60 that stores unified data from the unified processing unit 50 and generates and manages four-dimensional mapping information, as well as a display unit 70 (see Figure 8). In addition, some or all of the CV calculation unit 40, unified processing unit 50, four-dimensional mapping information generation unit 60, and display unit 70 can be provided separately from the exploration vehicle 10. Specifically, the CV calculation unit 40, the unification processing unit 50, the four-dimensional mapping information generation unit 60, and the display unit 70 are composed of information processing devices such as a PC on which predetermined software (programs) is implemented.

[0022] [3D Ground Penetrating Radar 20] The three-dimensional ground-penetrating radar 20 installed on the exploration vehicle 10 is a ground-penetrating exploration means (sensor) for generating three-dimensional underground information beneath the road surface on which the exploration vehicle 10 travels. As shown in Figure 3, it consists of a radar unit 21 mounted inside the exploration vehicle 10 and an antenna 22 positioned on the bottom surface of the exploration vehicle 10. The three-dimensional ground-penetrating radar 20 according to this embodiment is a geophysical exploration radar that explores and detects the presence of underground cavities, buried pipes, etc. by utilizing the phenomenon of electromagnetic waves being reflected at physical boundaries in the ground. By controlling the radar body 21 to transmit and receive electromagnetic waves from the antenna 22 while driving the exploration vehicle 10 at a predetermined speed (for example, 60 km / h), it is possible to easily map information down to, for example, 2 meters underground in three dimensions (see Figure 5 described later).

[0023] Specifically, the three-dimensional ground-penetrating radar 20 is a mobile underground CT scanner that utilizes electromagnetic waves. For example, it uses a ground-penetrating radar antenna (multi-channel) with up to 36 channels manufactured by Kontur of Norway, enabling it to visualize the underground environment in 3D while the exploration vehicle 10 is moving. In contrast, conventional ground-penetrating radars, such as those used in hand-operated sensors, were single-channel. A single-channel antenna system consists of a pair of sensors for transmitting and receiving electromagnetic waves.

[0024] In contrast, the multi-channel system employed in the three-dimensional ground-penetrating radar 20 of this embodiment is equipped with multiple sensors (transmitting and receiving means) consisting of multiple transmitting and receiving units within the antenna, and can acquire multiple two-dimensional cross-sections, thereby enabling the exploration and generation of underground conditions as three-dimensional information. In other words, the three-dimensional ground-penetrating radar 20 arranges multiple sensors, each consisting of a pair of electromagnetic wave transmitting and receiving units, and superimposes two-dimensional cross-sections to create a three-dimensional representation of the underground, similar to a CT scan. For example, underground cavities can be detected independently, while buried pipes can be detected continuously. Conventionally, multiple single-channel antennas were arranged in a row to acquire two-dimensional cross-sectional information at, for example, a pitch of approximately 0.5 m. In contrast, the multi-channel antenna according to this embodiment can be densely arranged at, for example, a pitch of 7.5 cm, to acquire high-resolution three-dimensional information about the underground conditions.

[0025] In this embodiment, the antenna 22 of the three-dimensional ground-penetrating radar 20 is positioned between the front and rear wheels on the bottom surface of the exploration vehicle 10, or between the axles of multiple wheels, as shown in Figure 3(a). By positioning the antenna 22 of the three-dimensional ground-penetrating radar 20 on the bottom surface of the exploration vehicle 10, as shown in Figure 3(b), ground-penetrating exploration data can be acquired across the entire width of the road surface, even on roads exceeding the electromagnetic wave irradiation range (irradiation width) of the antenna 22, by driving the exploration vehicle 10 over the same road surface multiple times (for example, twice on the same road) while shifting in the width direction of the vehicle. Figure 3(b) shows a case where ground-penetrating exploration data at 7.5 cm intervals can be acquired over the entire width of a road, for example, 3.0 m, by driving the exploration vehicle 10 over the same road surface twice.

[0026] In this context, the configuration in which the antenna is positioned on the underside of the vehicle is called the "ground type," and it has many advantages compared to the conventional "air type," in which the antenna protrudes significantly from the front or rear of the vehicle, or a cargo bed or similar structure with the antenna mounted on it is towed. The air-type radar is designed for use in mine clearance in unpaved areas, and its antenna is positioned approximately 20-30 cm above the ground for measurement. While this makes it easy to mount on vehicles (e.g., by attaching it to the front of the vehicle or towing it), the large distance between the antenna and the ground results in attenuation of electromagnetic waves, leading to a shallower detection depth. Thus, conventional air-type antennas, due to their antenna characteristics requiring them to be suspended 20-30 cm above the ground, and their thickness of approximately 20 cm, could not be used as vehicle-mounted (ground-type) antennas that were directly installed on vehicles.

[0027] In contrast, the ground-type radar is designed for use at close range from the road surface, specifically allowing measurements to be taken at approximately 3 to 10 cm above the ground. Therefore, electromagnetic waves are less attenuated, and the exploration depth is about 1.5 to 2 times deeper compared to air-type probes. Furthermore, the ground-type antenna can be positioned as close to the ground surface as possible, and because it is mounted directly on the vehicle, the antenna height is also thin, at about 10 cm, resulting in superior stability and safety for measurements during ground-penetrating exploration.

[0028] In this embodiment, a ground-type system with the above-mentioned advantages is adopted, and the antenna 22 of the three-dimensional ground-penetrating radar 20, which serves as a ground-penetrating exploration means, is positioned on the bottom surface of the exploration vehicle 10. Specifically, for example, the aforementioned ground-type antenna manufactured by Kontur of Norway is used, and the antenna 22 of the three-dimensional ground-penetrating radar 20 is mounted between the axles (between the wheelbases) of the front and rear wheels of the exploration vehicle 10. Kontur's ground-type antennas are more than half the height of conventional air-type antennas and ground-type antennas from other companies, allowing them to be installed in the most stable position between the front and rear axles of vehicles.

[0029] By installing the antenna 22 of the three-dimensional ground-penetrating radar 20 between the wheelbases of the exploration vehicle 10, the following advantages are obtained. • By shortening the overall length of the exploration vehicle 10, the difference in turning radius between the inner and outer wheels can be reduced. • By shortening the overall length of exploration vehicle 10, it becomes more maneuverable. • Unlike towed vehicles, there's no need to worry about what's behind you. • Reversing becomes easier compared to the towed type. Compared to rear-wheel mounting types, the antenna is at a consistent height relative to the road surface, allowing for stable data acquisition.

[0030] For ground-penetrating radar surveys, a constant antenna height is extremely important for data accuracy, and the advantages of positioning the antenna 22 of the three-dimensional ground-penetrating radar 20 between the wheelbases of the survey vehicle 10 are significant. The antenna 22, positioned between the wheelbases of the exploration vehicle 10, is placed at an optimal position in the vehicle's length direction, either between the front and rear wheels of the exploration vehicle 10, or between multiple wheels. In other words, the antenna 22 is positioned at an appropriate and optimal location in the vehicle's length direction (front and rear of the vehicle's direction of travel) depending on the vehicle's length, the arrangement of the front and rear wheels, the radar's output and characteristics, etc. Therefore, it is preferable that the antenna 22 be configured to be movable and adjustable in the longitudinal direction of the exploration vehicle 10.

[0031] Furthermore, in this embodiment, the antenna 22 of the three-dimensional ground-penetrating radar 20 is configured to be movable and adjustable in the vehicle width direction (left and right in the direction of vehicle travel) between the wheelbases of the exploration vehicle 10. Specifically, as shown in Figures 4(a) and 4(b), the antenna 22 of the three-dimensional ground-penetrating radar 20 can be slid left and right in the width direction of the exploration vehicle 10, and can be fixed at a desired position in the width direction. This allows the antenna 22 to be moved and adjusted according to the size (road width) of the road surface to be explored, enabling it to accommodate road surfaces of various sizes (widths).

[0032] In the example shown in Figure 4, by moving the antenna 22, which has a survey width of 1.8 m, so that it protrudes 15 cm from each of the left and right sides of the survey vehicle 10, ground survey data can be acquired for a road surface with a width of 3.5 m in two passes (b). In this way, by making the antenna 22 of the three-dimensional ground-penetrating radar 20 movable in the vehicle width direction between the wheelbases of the exploration vehicle 10, it is possible to acquire data over a range exceeding the antenna width (for example, up to 3.5m) by sliding one antenna (for example, 1.8m) to the left or right of the exploration vehicle and driving it multiple times (for example, twice) (see Figure 4). Furthermore, as shown in Figure 4, by limiting the protrusion of the exploration vehicle 10 to approximately 15 cm on each side, the risk of collisions with roadside obstacles or third parties can be reduced or avoided compared to, for example, a 2.5 m wide antenna used in conventional air-type antennas.

[0033] Furthermore, the range of movement of the antenna 22 in the vehicle width direction can be arbitrarily set according to the antenna width, the vehicle width of the exploration vehicle 10, etc. Furthermore, the movable structure of the antenna 22 is not particularly limited, as long as the antenna 22 can be moved in the vehicle width direction, such as by a sliding structure or a stepped structure with multiple fixing positions using bolts, etc. Furthermore, while the antenna 22 can be positioned exposed on the bottom surface of the exploration vehicle 10, it can also be covered in whole or in part with a cover to protect against damage from bouncing or impacting foreign objects, contact with the ground or obstacles, etc. For example, a plastic cover that does not affect the transmission and reception of electromagnetic waves by the antenna 22 can be provided.

[0034] Figure 5 shows the ground-penetrating exploration data acquired and generated by the three-dimensional ground-penetrating radar 20 according to this embodiment. As shown in Figure 5(a), the three-dimensional ground-penetrating radar 20 moves along with the exploration vehicle 10, and the multi-channel antenna 22 acquires two-dimensional underground information at predetermined intervals (e.g., 7.5 cm pitch) along the direction of vehicle movement, and acquires and generates three-dimensional underground information. As shown in Figure 5(b), this three-dimensional underground information represents three-dimensional underground cross-sectional information of vertical cross-sections (antenna direction), vertical cross-sections (direction of travel), and horizontal cross-sections, based on underground data in the direction of antenna 22 arrangement (vehicle width direction) and the direction of antenna 22 travel (vehicle length direction). By generating images from these, it is possible to output three-dimensional underground image information. The generation and output of this underground three-dimensional information is performed by the radar unit 21 of the three-dimensional ground-penetrating radar 20, or by a PC connected to the radar unit 21.

[0035] [30-degree camera] The omnidirectional camera 30 is positioned, for example, on the roof of the exploration vehicle 10 and is a ground image generation means that generates three-dimensional ground images of the road surface on which the exploration vehicle 10 travels. Specifically, the omnidirectional camera 30 is one or more video cameras capable of capturing 360° images around the moving exploration vehicle 10, and is configured to capture three-dimensional images (all-around images) of the ground corresponding to the ground-penetrating data acquired by the three-dimensional ground-penetrating radar 20 as the exploration vehicle 10 moves.

[0036] Figure 6 shows an example of a 360-degree stereoscopic panoramic image of the ground acquired and generated by the omnidirectional camera 30 of this embodiment. The omnidirectional camera 30 can capture a 360° panoramic image of its surroundings. For example, as shown in Figure 6(a), it captures the ground conditions and scenery within a radius of approximately 5m centered on the exploration vehicle 10 traveling on the road surface. In the example shown in the figure, it can be seen that a 360° panoramic image can be captured and acquired for a road or bridge with a total width of 14.55m and an effective width of 13.75m, centered on the exploration vehicle 10 traveling in the center of one lane, with a roadway area of ​​3.5m, shoulder of 0.5m, side strip of 0.5m, sidewalk area of ​​2.0m, median strip of 0.75m, and curb of 0.4m. Furthermore, as shown in Figure 6(b), the ground conditions and scenery are imaged within a radius of approximately 5 to 25 m centered on the exploration vehicle 10 traveling on the road surface, and 360° panoramic images can also be captured and acquired of the slopes (embankments) next to the road surface. Figure 6(c) shows an example of the output of a 360° panoramic image captured by the omnidirectional camera 30.

[0037] In this embodiment, by performing predetermined camera vector calculations based on the 360° panoramic image acquired and generated by the omnidirectional camera 30, high-precision three-dimensional ground mapping can be easily performed simply by attaching the omnidirectional camera 30 to the exploration vehicle 10 and driving it (for example, at a speed of about 60 km / h), without performing on-site surveys or relying on GPS location information. Generally, the positional information obtained from GPS and four-way cameras mounted on vehicles that make up a survey vehicle can have an error of, for example, 0.5m to several meters depending on the GPS reception conditions. Therefore, it is difficult to pinpoint the location of any specific anomaly using only this information. Furthermore, GPS cannot be received inside tunnels or under overpasses, so the only positional information available is the image from the vehicle's four-way cameras, making it even more difficult to pinpoint the location of any specific area. Therefore, in this embodiment, based on the 360° panoramic image acquired by the omnidirectional camera 30, the camera vector calculation unit 40 performs predetermined camera vector calculations, thereby enabling high-precision three-dimensional ground mapping without relying on GPS or performing on-site surveying and measurement.

[0038] [CV calculation section 40] The CV (camera vector) calculation unit 40 is a camera vector calculation means that can calculate and generate highly accurate positional information from image data of a three-dimensional ground image captured by the omnidirectional camera 30 described above. Specifically, the CV calculation unit 40 according to this embodiment consists of means such as a feature point extraction unit 41 that automatically extracts a predetermined number of feature points, a feature point correspondence processing unit 42 that automatically tracks the extracted feature points within each frame image of the video footage and determines the correspondence between frame images, and a camera vector calculation unit 43 that determines the three-dimensional position coordinates of the feature points for which the correspondence has been determined and determines a camera vector consisting of the three-dimensional position coordinates and three-dimensional rotation coordinates of the camera corresponding to each frame image from said three-dimensional position coordinates (see Figure 12 described later). Camera vector calculation is a technology disclosed in the applicant's patent publication No. 6446005, etc., which automatically extracts feature points from a 360° panoramic video, tracks them across multiple adjacent frames, constructs a triangle calibrated between the camera movement base and the tracking points, and analyzes the tracking data to obtain a panoramic CV (camera vector) video that contains the three-dimensional coordinates of the feature points and the camera position and orientation.

[0039] The CV (camera vector) values ​​obtained through CV calculation represent the position and orientation of the camera (omnidirectional camera 30) in terms of 6 degrees of freedom. In other words, for all frames of the 360° panoramic video captured by the omnidirectional camera 30, the coefficients for each of the 6 degrees of freedom, x, y, z, θx, θy, and θz, are determined by calculation. This allows relative positional information and three-dimensional coordinate information to be generated and added to the 360° video solely through calculation, without relying on positional information from GPS or surveying. This allows for the generation and assignment of CV values ​​for any 360-degree video, and based on these CV values, a 3D information unification process is performed that unifies the 360-degree video with the 3D underground information acquired by the 3D ground-penetrating radar 20 described above. The specific details of the CV calculation in this CV calculation unit 40 will be described later with reference to Figures 12 to 22.

[0040] [Unified Processing Unit 50] The unification processing unit 50 is an unification processing means that generates unified three-dimensional information from underground to above ground space by performing a predetermined data unification process on the three-dimensional underground information generated by the three-dimensional ground-penetrating radar 20 and the three-dimensional ground image generated by the panoramic camera 30. Specifically, the unification processing unit 50 according to this embodiment unifies the underground three-dimensional information and the ground three-dimensional image based on the camera vector generated by the CV calculation unit 40 and the position coordinate information contained in the underground three-dimensional information acquired by the three-dimensional ground-penetrating radar 20.

[0041] Figure 7 is a flowchart showing the procedure for unifying ground-penetrating exploration data and ground image data by the unification processing unit 50 of this embodiment. As shown in the figure, in the unification process of this embodiment, first, ground-penetrating exploration data acquired by the three-dimensional ground-penetrating radar 20 (Step 1: S1) and 360° panoramic images of the ground acquired by the omnidirectional camera 30 (Step 2: S2) are input to the unification processing unit 50. At this time, the ground-penetrating survey data acquired by the three-dimensional ground-penetrating radar 20 includes location information based on position information and time information acquired by the GPS (high-precision GNSS) 11 mounted on the survey vehicle 10. The 360° panoramic image from the omnidirectional camera 30 includes positional information based on location and time information acquired by the GPS (high-precision GNSS) 11 mounted on the exploration vehicle 10, and is also processed by the CV calculation unit 40 (step 3: S3) to assign CV values ​​that represent three-dimensional position coordinates.

[0042] Then, these ground-penetrating survey data and 360° panoramic images are linked based on their corresponding positional information, and the CV values ​​calculated and assigned by the CV calculation unit 40 are assigned to the 360° panoramic images and the corresponding ground-penetrating survey data, linking them by the same positional coordinates (Step 4: S4). As a result, the ground-penetrating exploration data becomes highly accurate mapped three-dimensional underground information with CV values ​​assigned as position coordinates (Step 5: S5). Furthermore, the 360° panoramic video is processed with orthorectification to generate a three-dimensional terrestrial image that includes a high-precision orthorectified image with CV values ​​(Step 6: S6).

[0043] While known image generation techniques can be used to generate orthomosaic images, in this embodiment, high-precision orthomosaic images can be generated and output based on a 360-degree video with assigned CV values. Specifically, the CV value is used as positional information to convert a 360° panoramic image into a distortion-free orthographic projection image as if viewed from directly above, and multiple orthographic images are joined (mosaiced) so that the seams are not noticeable to generate a single integrated orthoimage. This allows for the generation and storage of orthoimage data divided into desired ranges and sizes. Subsequently, based on the corresponding CV values, the subsurface 3D information and the ground 3D image including orthophotos are unified and processed, and generated and output as unified subsurface and ground data (Step 7: S7).

[0044] Through the high-precision 3D mapping achieved by unifying underground 3D information and above-ground 3D video as described above, the 3D position coordinates of underground and above-ground data can be acquired with an error of approximately 5 to 15 cm based on CV calculations, without being affected by GPS reception conditions, etc. Furthermore, orthomosaic images are created by converting a surface into an undistorted image, as if viewed from directly above, and adding positional information. Orthomosaic images of road surfaces can be generated from images captured by omnidirectional cameras with added three-dimensional coordinate data. Because such orthomosaic images have the same accuracy as planar maps, they are suitable for displaying three-dimensional ground-penetrating radar results.

[0045] [Four-dimensional mapping information generation unit 60] The four-dimensional mapping information generation unit 60 is a means for generating four-dimensional information by adding time axis information indicating the generation time of each piece of three-dimensional information to the unified three-dimensional information of underground and above ground generated by the unification processing unit 50. Figure 8 is a block diagram showing the basic configuration of the four-dimensional mapping information generation unit 60 of this embodiment. As shown in the figure, the four-dimensional mapping information generation unit 60 is configured to include a unified information storage unit 61, a unified information extraction unit 62, and a comparison image generation unit 63.

[0046] [Centralized Information Storage Unit 61] The unified information storage unit 61 is a unified information storage means that stores multiple unified information items generated by the unified processing unit 50 at different times, by adding time axis information indicating the time of generation. Specifically, the unified information storage unit 61 is a storage means that serves as a database for storing and managing the unified information generated by the unified processing unit 50. It stores and manages multiple unified information items in chronological order based on time axis information, and allows for the arbitrary extraction and referencing of necessary information. The time axis information attached to the multiple unified information stored and managed by the unified information storage unit 61 consists of time information indicating the date and time each unified information was generated, or the date and time stored and remembered in the unified information storage unit 61.

[0047] This unified information storage unit 61 allows for the storage and management of multiple unified pieces of information that are generated at different times but located at the same ground level, such as unified pieces of information generated during normal times and unified pieces of information generated during disasters. This will allow for the extraction and referencing of ground conditions before and after a natural disaster, based on centralized information generated during normal times and centralized information acquired after the disaster, making it useful information for early recovery and reconstruction.

[0048] Furthermore, the unified information stored in the unified information storage unit 61 can be linked and stored, and managed, with location information and time information, as well as any related and corresponding information. For example, the unified information can be linked and stored with any useful related information, such as the name, address, telephone number, owner, attributes, and other information of buildings and structures on the ground that correspond to the location information. As a result, the unified information storage unit 61 can function as a platform for four-dimensional mapping information, where unified four-dimensional information from underground and above ground, along with useful related information, is stored. Furthermore, by accessing such a four-dimensional mapping information platform and enabling the extraction and referencing of information, it becomes possible to utilize this information as useful data for infrastructure development and pre-disaster recovery during peacetime, as well as for recovery and reconstruction support during disasters.

[0049] [Unified information extraction unit 62] The unified information extraction unit 62 is an unified information extraction means that extracts multiple unified pieces of information from the multiple unified pieces of information stored in the unified information storage unit 61 described above, based on subterranean three-dimensional information, that are generated at different times but have the same above-ground location. Specifically, the unified information extraction unit 62 can extract multiple unified pieces of information that are generated at different times but have the same above-ground location, based on the three-dimensional positional information of underground cavities and / or buried objects.

[0050] As described above, the unified information generated and processed by the unified processing unit 50 is linked to the underground three-dimensional information and the above-ground three-dimensional image by highly accurate mapped position information, to which CV values ​​are given as common position coordinates (see Figure 7). Furthermore, three-dimensional information about the underground, in particular, has the characteristic of being superior in "robustness," meaning that the position of underground structures and buried objects is less likely to change compared to buildings and structures above ground. This is equivalent to a fingerprint in the human body and can be considered the "fingerprint of the Earth." For example, even if above-ground structures are washed away, moved, or destroyed by a tsunami or fire, the three-dimensional position of buried objects and underground structures will not change much even if a large-scale earthquake occurs. Even if there is some movement, the relative positions of each buried object and underground structure will not change much, and the relative positional relationships will remain stable.

[0051] Therefore, the unified information extraction unit 62 searches and references multiple unified information sources and extracts unified information sources whose three-dimensional positional information of underground cavities and / or buried objects contained in each unified information source matches or approximates, as unified information sources with different generation times but matching above-ground positions. This means that if three-dimensional images of the underground are acquired in advance, in the event of a disaster, the three-dimensional positional information of buried objects underground can be used as benchmark and mapping information, and the three-dimensional images of the surface linked to that underground three-dimensional information can be extracted and identified, making it possible to compare and refer to surface images from normal times and during a disaster. Furthermore, because the unified underground and above-ground information is linked by relative coordinates through the aforementioned CV calculation, even if there is a shift in the absolute coordinates of the reference point during a disaster, the exact location can be quickly determined.

[0052] Furthermore, the process of identifying and extracting terrestrial 3D images based on subsurface 3D information can be performed not only by matching and approximating the 3D positional information of subsurface cavities and buried objects, but also, instead of identifying based on subsurface 3D positional information, by comparing patterns of multiple subsurface 3D images (see Figure 5(b)) or terrestrial 3D images, for example, by visual inspection, manual comparison, or algorithmic pattern comparison. The identification and extraction of ground-level three-dimensional images based on underground three-dimensional information by the unified information extraction unit 62 will be explained in the following embodiment with reference to the drawings, including cases where ground-level reference points move or disappear, and there are positional changes in underground buried objects, etc.

[0053] [Comparison image generation unit 63] The comparison video generation unit 63 is a comparison video generation means that outputs multiple ground-based three-dimensional images that are generated at different times but have the same ground position, from multiple unified information extracted and identified by the unified information extraction unit 62, in a way that allows for comparison. Specifically, the comparison image generation unit 63 generates multiple comparison images with the same ground position but different generation times, and displays them on a display unit 70 (see Figure 8), which consists of a predetermined display device, such as a screen. The comparison video generated and output by this comparison video generation unit 63 is displayed in a way that allows for comparison, for example, as images of the same location during normal times and after a disaster, as shown in Figure 11. Furthermore, such comparative images can be printed using a printer or other printing device, and then processed and edited as desired using an information processing device such as a PC.

[0054] Figures 9 to 11 show specific examples of the four-dimensional mapping information generated and output by the four-dimensional mapping information generation system according to this embodiment, including images generated and output by the comparison image generation unit 63. Figure 9 shows an example of subsurface 3D data and an image obtained by unifying the subsurface 3D data into a surface orthomosaic image. Figure 9(a) shows the output result of unifying underground information of cavities and buried pipes obtained by the three-dimensional ground-penetrating radar 20 and above-ground information such as road surface cracks into an orthomosaic image of the road surface generated from three-dimensional positional information obtained by the CV calculation unit 40. The road surface cracks and subsidence conditions shown in the figure can be read by image analysis. Figure 9(b) shows the internal abnormalities (reinforcement corrosion) of the bridge deck detected by the three-dimensional ground-penetrating radar 20, along with an orthomosaic image of the road surface generated from the three-dimensional positional information obtained by the CV calculation unit 40.

[0055] Figure 10 is an example of an image created by unifying underground 3D data, surface orthomosaic images, real-time surface video, and map video. As shown in the figure, the ground-penetrating exploration data and three-dimensional underground information acquired by the sensors of the ground-penetrating exploration device (upper left: see Figure 5(b)), the orthomosaic image and road surface vertical image based on the ground-level three-dimensional image integrated with this ground-penetrating exploration data (lower left: see Figure 9), the 360-degree panoramic camera image taken by the exploration vehicle 10 (upper right: see Figure 6(c)), and the corresponding map image (lower right) can be generated and output as a single, comparable image.

[0056] Figure 11 shows an example of four-dimensional mapping information, comparing images of the same location during normal times and after a disaster. As shown in the figure, from unified three-dimensional information of underground and above ground, it is possible to generate and output a single, comparable image of a 360-degree panoramic view of the ground, with the same ground position but generated at different times, based on three-dimensional underground information such as buried objects and cavities. Furthermore, the example of generated images shown in Figures 9 to 11 can be displayed on a predetermined display device, such as the display unit 70 described above, and can also be printed using a printer or other printing device. Needless to say, they can also be arbitrarily processed and edited using an information processing device such as a PC.

[0057] The output image, which displays unified three-dimensional information of both underground and above ground, clearly identifies the location of rebar corrosion. Furthermore, by extracting information and images generated at different times based on the time axis information, it is possible to visually compare the time-series data changes between normal and disaster situations after the next survey or repair work, and to create a database of necessary information. Furthermore, creating a database of abnormal locations provides useful information for purposes such as pre-disaster recovery and repair planning, as well as for the maintenance and management of infrastructure facilities.

[0058] Furthermore, it is possible to link and store any relevant information in a four-dimensional database that combines unified three-dimensional information of underground and above-ground data with time-axis information, and to access necessary information via the internet, for example. This will enable the provision of a four-dimensional mapping information platform that allows users to search and refer to useful and effective related information, including unified underground and above-ground data.

[0059] By unifying the subsurface three-dimensional data and surface three-dimensional images obtained in this embodiment, and by generating and managing four-dimensional mapping information to which time axis information is added, the following effects can be obtained. (1) Because high-precision three-dimensional coordinates can be obtained, the position determination using two-dimensional ground-penetrating radar, which is used in conventional ground-penetrating exploration, becomes unnecessary. (2) The position correction device makes it possible to obtain high-precision three-dimensional coordinates even in places where GPS signals cannot be received, such as tunnels and elevated bridges. (3) Since high-precision three-dimensional coordinates of the ground can be obtained efficiently, there is no need to measure the shape of structures or surrounding objects on site. (4) Underground information from the three-dimensional ground-penetrating radar and ground information from the 360-degree camera 30 and CV calculation unit 40 can be centrally managed by creating a database. (5) Visual information from high-precision orthomosaic images allows for time-series management of information, such as comparisons with the next survey or after repair work. (6) Orthomosaic images offer the same level of accuracy as planar maps, but are visually superior to maps, thus providing effective results to clients. (7) The efficiency and centralization of surveys, along with the ability to easily obtain highly accurate three-dimensional coordinate information from video footage, will enable a rapid response in emergencies, such as ensuring the safety of supply roads during disasters. (8) By using unified information with added time axis information, it is possible to extract and identify multiple unified pieces of information that are generated at different times but have the same ground location, making it possible to compare and refer to real ground images from normal times and after a disaster. This allows for the rapid provision of information useful for the recovery and reconstruction of disaster-stricken areas without relying on ground survey control points or structures.

[0060] [CV calculation processing] Next, the specific details of the CV calculation performed by the CV calculation unit 40 of the ground-penetrating exploration device, which constitutes the four-dimensional mapping information system according to the above embodiment, will be explained with reference to Figures 12 to 22. As described above, CV calculation is one method for determining CV values, and the results obtained through CV calculation are called CV values ​​or CV data. The notation CV is an abbreviation for Camera Vector, and a camera vector (CV) is a value that indicates the three-dimensional position and 3-axis rotational orientation of a camera, such as a video camera, that acquires images for measurement purposes. CV calculation involves acquiring moving images (video footage), detecting feature points within that footage, tracking them across multiple adjacent frames, generating numerous triangles within the image formed by the camera position and the tracking trajectories of the feature points, and then analyzing these triangles to determine the camera's three-dimensional position and its 3-axis rotational orientation.

[0061] A key characteristic of CV calculation is that, in the process of determining CV values, the three-dimensional coordinates of feature points (reference points) within the image are simultaneously determined. Furthermore, the CV values ​​calculated from the video image simultaneously determine the three-dimensional camera position and three-dimensional camera orientation for each frame of the video. Moreover, the ability to calculate CV values ​​in conjunction with the video using only one camera is a superior feature that can only be achieved through CV calculation. However, CV values ​​can also be obtained from other measurement methods (such as GPS and gyroscope, or IMU). Therefore, obtaining CV values ​​is not limited to CV calculation. Furthermore, when acquiring CV values ​​using GPS, a gyroscope, or an IMU, it is necessary to synchronize the image frame and the measurement sampling time with high precision and complete accuracy in order to simultaneously acquire each frame of the moving image, along with its three-dimensional camera position and three-dimensional camera orientation.

[0062] CV data calculated from video images is relative before processing, but for short intervals, it can acquire highly accurate three-dimensional position information and 3-axis rotation angle information. Furthermore, when acquiring CV data from an image, the acquired data is relative, and calibration at a known point is necessary to convert it to an absolute value. However, it possesses a superior characteristic that is difficult to achieve by other methods: the positional relationship with any object in the image can be measured using the three-dimensional position coordinates and orientation of feature points acquired simultaneously with the CV values, or by newly designated and acquired feature points within the image. Furthermore, since CV values ​​corresponding to the image are obtained, it is highly compatible with images, and as long as the CV values ​​are obtained from an image, there are no inconsistencies within that image. Therefore, by comparing them with CV values ​​obtained from sources other than the image, CV calculations that can directly determine the camera position and its 3-axis rotational orientation from the image are suitable for in-image measurement and in-image surveying. Furthermore, in this invention, based on the CV value data obtained by this CV calculation, the above-mentioned ground orthomosaic image is generated, and an integrated image of the subsurface three-dimensional data and the ground orthomosaic image is generated.

[0063] [CV calculation section] The CV calculation unit 40 obtains CV values ​​by performing predetermined CV calculation processing on video images captured by the omnidirectional camera 30 described above. Specifically, as shown in Figure 12, it comprises a feature point extraction unit 41, a feature point correspondence processing unit 42, a camera vector calculation unit 43, an error minimization unit 44, a three-dimensional information tracking unit 45, and a high-precision camera vector calculation unit 46. First, in principle, any video can be used for CV calculations. However, with videos that have a limited field of view, the image will be interrupted when the viewpoint direction changes. Therefore, in practice, it is desirable to use panoramic video (see Figures 6(c) and 13) or to treat wide-angle video as part of a panoramic video. Note that video is similar to a series of still images and can be treated as multiple still images. Furthermore, while pre-recorded video footage is generally used, it is also possible to use real-time video footage captured in accordance with the movement of moving objects such as automobiles.

[0064] Therefore, in this embodiment, as the video used for CV calculation, a 360-degree panoramic video (see Figures 6(c) and 13) capturing the entire 360-degree surroundings of a moving object such as a vehicle, or a wide-angle video close to a panoramic video, can be used, and by unfolding the panoramic video in the direction of the viewpoint, it can be treated as part of the panoramic video. The planar unfolding of a 360-degree image is the process of representing the 360-degree image as a regular image using perspective projection. The term "perspective projection" is used here because the 360-degree image itself is displayed using methods other than perspective projection, such as Mercator projection or spherical projection (see Figure 13). By unfolding it into a planar image, such as using an equidistant projection, it can be converted and displayed as a regular perspective image.

[0065] In this embodiment, first, the exploration vehicle 10 is driven, and in order to acquire CV value data, an omnidirectional camera 30 fixed to the exploration vehicle 10 captures a 360° panoramic image of the area around the exploration vehicle 10 as the exploration vehicle 10 moves (see Figures 6(c) and 13). Furthermore, the exploration vehicle 10 may be equipped with position measurement equipment, such as a GPS device alone or a GPS device with an IMU attached, for the purpose of acquiring its position coordinates. Furthermore, the omnidirectional camera 30 mounted on the exploration vehicle 10 can be any configuration as long as it is a camera that can capture and acquire wide-area images. Examples include cameras with wide-angle lenses or fisheye lenses, mobile cameras, fixed cameras, cameras with multiple fixed cameras, and cameras that can rotate 360 ​​degrees. In this embodiment, as described above, one or more cameras are integrally fixed to the exploration vehicle 10, and an omnidirectional camera 30 is used that captures wide-area images as the exploration vehicle 10 moves.

[0066] Furthermore, with the omnidirectional camera 30 described above, by installing it on the roof of the exploration vehicle 10, it is possible to simultaneously capture 360-degree images of the surrounding area with one or more cameras, and as the exploration vehicle 10 drives and moves, wide-area images can be acquired as video data. Here, the omnidirectional camera 30 is a video camera that can directly acquire images of the entire circumference of the camera, but if it can acquire images of more than half of the camera's entire circumference, it can be used as omnidirectional images. Furthermore, even with a standard camera with a limited field of view, it is possible to treat it as part of a 360-degree image, although the accuracy of the CV calculation will be reduced.

[0067] Furthermore, the wide-angle video captured by the omnidirectional camera 30 can be superimposed as a single image onto a virtual sphere that matches the field of view at the time of capture. Spherical image data superimposed onto a virtual sphere is saved and output as spherical image (360-degree image) data in the state it is superimposed onto the virtual sphere. The virtual sphere can be set to any spherical shape with the camera unit that acquires wide-area images as its center point. Figure 13(a) shows an image of the appearance of a virtual sphere onto which a spherical image is superimposed, and Figure 13(b) shows an example of a spherical image superimposed on the virtual sphere. Figure 13(c) shows an example of an image obtained by unfolding the spherical image from (b) into a plane according to the Mercator projection.

[0068] The 360-degree video footage generated and acquired as described above is then input to the CV calculation unit 40 to obtain CV value data (see Figure 12). In the CV calculation unit 40, first, the feature point extraction unit 41 automatically extracts a sufficient number of feature points (reference points) from the video data captured and temporarily recorded by the omnidirectional camera 30. The feature point correspondence processing unit 42 automatically tracks the automatically extracted feature points within each frame image between frames to automatically determine their correspondence. The camera vector calculation unit 43 automatically calculates the camera vector corresponding to each frame image from the three-dimensional position coordinates of the feature points for which a correspondence has been determined. The error minimization unit 44 performs statistical processing on multiple camera positions through overlapping calculations to minimize the distribution of solutions for each camera vector, and automatically determines the camera position direction after the error minimization process.

[0069] The three-dimensional information tracking unit 45 positions the camera vector obtained by the camera vector calculation unit 43 as a rough camera vector, and then, based on the three-dimensional information obtained sequentially as part of the image in subsequent processes, automatically tracks the partial three-dimensional information contained in multiple frame images along the images of adjacent frames. Here, three-dimensional information (three-dimensional shape) mainly refers to the three-dimensional distribution information of feature points, that is, a collection of three-dimensional points, and this collection of three-dimensional points constitutes the three-dimensional shape. The high-precision camera vector calculation unit 46 generates and outputs a camera vector with even higher precision than the camera vector obtained by the camera vector calculation unit 43, based on the tracking data obtained by the three-dimensional information tracking unit 45. The camera vectors obtained in this manner are then input to the unification processing unit 50 described above and used for generating ground orthomosaic images, unifying subsurface three-dimensional data and ground orthomosaic images, and for extraction processing by the unification information extraction unit 62 of the four-dimensional mapping information generation unit 60.

[0070] There are several methods for detecting the camera vector from feature points of multiple images (video or sequential still images), but in the CV calculation unit 40 of this embodiment shown in Figure 12, a sufficiently large number of feature points are automatically extracted from the image and automatically tracked, and the three-dimensional vector and three-axis rotation vector of the camera are determined using triangulation geometry and epipolar geometry. By selecting a sufficiently large number of feature points, the camera vector information overlaps, allowing us to determine a unit triangle composed of the coordinates of three points from more than ten times that number of feature points. This minimizes errors from the overlapping information, enabling us to obtain a more accurate camera vector.

[0071] Camera vectors are the vectors representing the six degrees of freedom of a moving camera. Generally, a stationary three-dimensional object has six degrees of freedom: position coordinates (X, Y, Z) and rotation angles (Φx, Φy, Φz) for each of its coordinate axes. Therefore, the camera vector is the vector representing the camera's position coordinates (X, Y, Z) and the rotation angles (Φx, Φy, Φz) for each of its six degrees of freedom. If the camera is moving, the direction of movement is also included in the degrees of freedom, but this can be derived by differentiating from the six degrees of freedom mentioned above. Thus, in this embodiment, camera vector detection means that the camera takes six degrees of freedom for each frame, and determines six different coefficients for each frame.

[0072] The specific method for detecting camera vectors in the CV calculation unit 40 will be explained below with reference to Figure 14 and subsequent figures. First, the image data acquired by the omnidirectional camera 30 described above is input indirectly or directly to the feature point extraction unit 41 of the CV calculation unit 40. The feature point extraction unit 41 automatically extracts points or small region images that should be feature points from appropriately sampled frame images, and the feature point correspondence processing unit 42 automatically determines the correspondence between feature points across multiple frame images. Specifically, the system seeks to find a sufficient number of feature points to serve as the basis for detecting the camera vector. An example of feature points and their correspondence between images is shown in Figures 14-16. In the figures, "+" indicates an automatically extracted feature point, and the correspondence between multiple frame images is automatically tracked (see correspondence points 1-4 in Figure 16). Here, when extracting feature points, it is desirable to specify and extract a sufficient number of feature points in each image, as shown in Figure 17 (see the circles in Figure 17). For example, extracting around 100 feature points is recommended.

[0073] Next, the camera vector calculation unit 43 calculates the three-dimensional coordinates of the extracted feature points, and then calculates the camera vector based on those three-dimensional coordinates. Specifically, the camera vector calculation unit 43 continuously calculates the relative values ​​of various three-dimensional vectors, such as the positions of a sufficient number of features present between consecutive frames, the position vectors between moving cameras, the three-axis rotation vector of the camera, and the vectors connecting each camera position to the feature points. In this embodiment, for example, camera motion (camera position and camera rotation) is calculated by solving the epipolar equation from the epipolar geometry of a 360-degree panoramic image. Alternatively, this can be explained as triangulation, where a vehicle is equipped with a device that measures azimuth and elevation angles, and multiple target survey points are measured using the moving surveying device to obtain data on the same object, from which the coordinates of each target point and the trajectory of the surveying device's movement are determined.

[0074] Images 1 and 2 in Figure 16 are Mercator expansions of 360-degree panoramic images. If latitude is φ and longitude is θ, then the point on image 1 is (θ1, φ1) and the point on image 2 is (θ2, φ2). The spatial coordinates for each camera are z1=(cosφ1cosθ1,cosφ1sinθ1,sinφ1) and z2=(cosφ2cosθ2,cosφ2sinθ2,sinφ2). If the camera's translation vector is t and the camera's rotation matrix is ​​R, then the epipolar equation is z1T[t]×Rz2=0. By providing a sufficient number of feature points, t and R can be calculated as the least squares solution using linear algebra. This operation is then applied to multiple corresponding frames.

[0075] In this case, it is preferable to use a 360-degree panoramic image as the image used for calculating the camera vector. In principle, any image can be used for camera vector calculations, but 360-degree panoramic images are extremely advantageous because the direction of the object is directly represented by its latitude and longitude. Wide-angle images, such as the 360-degree panoramic image shown in Figure 16, allow for the selection of many feature points, making it easier to track them across multiple frames. Furthermore, even images taken with narrow-angle lenses are effective when capturing images from various directions while moving around the same object, as the feature points do not escape the field of view.

[0076] Therefore, in this embodiment, a 360-degree panoramic image is used for CV calculation. This allows for a longer tracking distance for feature points, enabling the selection of a sufficient number of feature points, and allowing for the selection of feature points suitable for long, medium, and short distances. Furthermore, when correcting rotation vectors, adding polar rotation transformation processing simplifies the calculation process. As a result, more accurate calculation results can be obtained. Note that Figure 16 shows a 360-degree spherical image, synthesized from images taken by one or more cameras, unfolded using the Mercator projection, a map projection method, in order to make the processing in the CV calculation unit 2 easier to understand. However, in actual CV calculations, it is not always necessary to use an unfolded image using the Mercator projection.

[0077] Next, the error minimization unit 44 calculates multiple vectors based on each feature point using multiple possible mathematical equations that arise from the number of camera positions and feature points corresponding to each frame. Statistical processing is then performed to minimize the distribution of the feature point positions and camera positions, thereby obtaining the final vector. For example, the Levenberg-Marquardt method is used to estimate the optimal least squares solution for the camera positions, camera rotations, and feature points of multiple frames, and the error is converged to obtain the camera positions, camera rotation matrices, and feature point coordinates. Furthermore, feature points with large error distributions are deleted, and recalculations are performed based on other feature points to improve the accuracy of calculations at each feature point and camera position. In this way, the position of feature points and the camera vector can be determined with high accuracy.

[0078] Figures 18 to 20 show examples of the three-dimensional coordinates of feature points obtained by CV calculation and the camera vector. Figures 18 to 20 are explanatory diagrams illustrating the vector detection method by CV calculation in this embodiment, and show the relative positional relationship between the camera and the object obtained from multiple frame images acquired by a moving camera. Figure 18 shows the three-dimensional coordinates of feature points 1-4 shown in images 1 and 2 of Figure 16, and the camera vector (X, Y, Z) moving between image 1 and image 2. Figures 19 and 20 show the positions of feature points obtained from a sufficient number of feature points and frame images, as well as the position of the moving camera. In these figures, the linearly continuous circles in the center of the graph represent the camera position, and the circles around them indicate the positions and heights of the feature points.

[0079] Here, in order to obtain more accurate three-dimensional information of feature points and camera positions at high speed, the CV calculation in the CV calculation unit 40 sets multiple feature points according to the distance from the camera to the feature points, as shown in Figure 21, and repeatedly performs multiple calculations. Specifically, the CV calculation unit 40 automatically detects feature points that have visual characteristics within the image, and when finding corresponding points for these feature points within each frame image, it focuses on the nth and n+mth frame images Fn and Fn+m used in camera vector calculations and performs unit calculations, repeating unit calculations with appropriately set n and m. m is the frame interval. Feature points are classified into multiple levels based on the distance from the camera to the feature point within the image. The further the distance from the camera to the feature point, the larger m becomes, and the closer the distance to the feature point, the smaller m becomes. This is because the further the distance from the camera to the feature point, the less the position changes between images.

[0080] Then, the classification of feature points based on their m values ​​is performed by setting multiple levels of m, ensuring sufficient overlap. As n progresses along with the image, the calculations are performed continuously. Furthermore, the same feature point is subjected to multiple duplicate calculations at each stage of m and as n progresses. In this way, by performing unit calculations focusing on frame images Fn and Fn+m, a precise camera vector can be calculated over a long period of time between each frame sampled every m frames (frames are dropped between frames), while a simplified calculation that can be performed in a short time can be used for the m frames between frame images Fn and Fn+m (the smallest unit frame).

[0081] Assuming there are no errors in the precise camera vector calculation for every m frames, the endpoints of the camera vectors for the m frames will coincide with the camera vectors Fn and Fn+m calculated with high precision. Therefore, the smallest unit of m frames between Fn and Fn+m can be calculated using a simplified method, and the scale of the m consecutive camera vectors can be adjusted so that the endpoints of the camera vectors of the smallest unit of m frames calculated using the simplified method coincide with the camera vectors Fn and Fn+m calculated with high precision. In this way, as n progresses continuously with the image, the camera vectors obtained by performing calculations multiple times on the same feature point are scaled and integrated to minimize the error, thereby determining the final camera vector. This allows for the acquisition of highly accurate camera vectors with no errors, while simultaneously speeding up the computation process by combining it with simplified calculations.

[0082] Here, there are various methods for simplified calculations depending on the required precision. For example, (1) while high-precision calculations use many feature points (100 or more), simplified calculations can use a minimum of about 10 feature points. Or, (2) even with the same number of feature points, if we consider the feature points and camera positions as equivalent, countless triangles are formed, and as many equations as there are triangles, the number of equations can be reduced to simplify the calculation. This process integrates the data by scaling it to minimize errors in each feature point and camera position, performs distance calculations, removes feature points with large error distributions, and recalculates for other feature points as needed, thereby improving the accuracy of calculations at each feature point and camera position.

[0083] Furthermore, performing these high-speed, simplified calculations enables real-time processing of camera vectors. Real-time processing of camera vectors involves performing calculations using the minimum number of frames and the minimum number of automatically extracted feature points required to achieve the desired accuracy. An approximate value of the camera vector is calculated and displayed in real time. As images accumulate, the number of frames and feature points are increased, allowing for more accurate camera vector calculations. The approximate value can then be replaced with a more accurate camera vector value for display.

[0084] Furthermore, in this embodiment, in order to obtain a more accurate camera vector, it is possible to track three-dimensional information (three-dimensional shape). Specifically, the three-dimensional information tracking unit 45 first positions the camera vector obtained via the camera vector calculation unit 43 and the error minimization unit 44 as a rough camera vector. Then, based on the three-dimensional information (three-dimensional shape) obtained as part of the image generated in subsequent processes, it automatically tracks the three-dimensional shape by continuously tracking the partial three-dimensional information contained in multiple frame images across adjacent frames. Then, from the tracking results of the three-dimensional information obtained by the three-dimensional information tracking unit 45, the high-precision camera vector calculation unit 46 calculates a more accurate camera vector.

[0085] The feature point extraction unit 41 and feature point matching processing unit 42 described above automatically track feature points within multiple interframe images, but the number of frames for feature point tracking may be limited due to the disappearance of feature points, etc. Furthermore, since images are two-dimensional and their shape changes during tracking, there are certain limitations to the tracking accuracy. Therefore, by positioning the camera vector obtained through feature point tracking as an approximate value, and then tracking the three-dimensional information (three-dimensional shape) obtained in subsequent processes on each frame image, a high-precision camera vector can be obtained from its trajectory. Tracking three-dimensional shapes makes it easier to achieve accurate matching and correlation. Since the three-dimensional shape and size do not change across frame images, tracking is possible over many frames, thereby improving the accuracy of camera vector calculations. This is possible because the approximate camera vectors are known by the camera vector calculation unit 43, and the three-dimensional shape is already known.

[0086] When camera vectors are approximate values, the error in three-dimensional coordinates across a very large number of frames accumulates over long distances because each frame is related to only a few frames through feature point tracking, resulting in a gradually larger error. However, the error in the three-dimensional shape when a portion of the image is cropped is relatively small, and its impact on shape changes and size is considerably small. For this reason, comparison and tracking using three-dimensional shape is extremely advantageous compared to two-dimensional shape tracking. In two-dimensional shape tracking, it is unavoidable to track changes in shape and size across multiple frames, leading to problems such as large errors or the inability to find corresponding points. However, in three-dimensional shape tracking, shape changes are extremely small, and in principle, there are no changes in size, making accurate tracking possible.

[0087] Here, the three-dimensional shape data to be tracked includes, for example, the three-dimensional distribution shape of feature points, or the polygon surface obtained from the three-dimensional distribution shape of feature points. Furthermore, the obtained three-dimensional shape can be converted into a two-dimensional image from the camera position and tracked as a two-dimensional image. Since the approximate values ​​of the camera vectors are known, it is possible to project the shape onto a two-dimensional image from the camera viewpoint, and it becomes possible to track changes in the shape of the object due to movement of the camera viewpoint.

[0088] The camera vectors obtained in the manner described above can be overlaid and displayed on the video footage captured by the omnidirectional camera 30. For example, as shown in Figure 22, the video from the in-vehicle camera is unfolded into a plane, corresponding points on the target plane are automatically searched for within each frame image, and the corresponding points are joined together to generate a combined image of the target plane. This combined image is then integrated into the same coordinate system and displayed. Furthermore, the camera position and orientation can be successively detected within this common coordinate system, and their positions, orientations, and trajectories can be plotted. The CV data shows the three-dimensional position and three-axis rotation, and by overlaying it on the video footage, the CV values ​​can be observed simultaneously in each frame of the video. An example of an image with CV data overlaid on video footage is shown in Figure 22. Furthermore, if the camera position is displayed correctly within the video footage, the position indicated by the CV value will be at the center of the image. If the camera movement is close to a straight line, the CV values ​​of all frames will overlap and be displayed. Therefore, as shown in Figure 22, for example, it is appropriate to deliberately display a position 1 meter directly below the camera position. Alternatively, it is more appropriate to display the CV value at the height of the road surface, using the distance to the road surface as a reference.

[0089] As explained above, the four-dimensional mapping information generation system of one embodiment of the present invention can achieve the following excellent effects. First, the unified underground and surface information that forms the basis of the four-dimensional mapping information is assigned high-precision three-dimensional coordinates of the surface based on CV calculations. This eliminates the need for position identification processes using hand-operated sensors, as in the conventional technology described above, and makes it possible to identify underground objects such as cavities and buried pipes using only orthomosaic images with unified underground data. In particular, conventional technology required, for example, four-way cameras to capture images of the front, rear, left, and right of the vehicle, as well as a simple positioning process based on GPS. Therefore, the benefits of this embodiment are significant.

[0090] Furthermore, because high-precision three-dimensional coordinates of the ground can be efficiently obtained through CV calculations, processes such as measuring the shape of structures or surrounding objects on-site for detection location become unnecessary, resulting in a significant improvement in work efficiency. Therefore, even if, for example, a large-scale disaster causes damage, movement, or loss of ground structures, survey control points, etc., accurate location information can be obtained based on normal three-dimensional ground images.

[0091] Furthermore, according to this embodiment, it is possible to continuously acquire omnidirectional information on underground and surface information that constitute the exploration target and area, and to centrally manage omnidirectional three-dimensional information of underground and surface. Furthermore, a centrally managed database of underground and above-ground information can be made compatible with GIS (Geographic Information System) related software (applications), such as "ArcGIS" (registered trademark) from Esri Corporation in the United States, enabling database management with excellent usability, versatility, and expandability. In this way, by accumulating and managing all-around three-dimensional information from underground and above ground in a database, it is possible to acquire and generate underground three-dimensional information and above-ground three-dimensional images over as wide an area as possible during normal times, for example, by driving on roads throughout Japan, and store them in the database. This can then be prepared as useful pre-disaster recovery information in the event of a large-scale earthquake or other disaster.

[0092] Furthermore, the three-dimensional underground and above-ground information generated and output in this embodiment can be stored and managed as four-dimensional information to which time axis information is added according to the time series of when the information was generated. This allows for the easy and rapid management and analysis of useful information necessary for recovery support and infrastructure development, such as comparing real-time ground images from normal times and after a large-scale disaster, identifying future survey points, and comparing and aligning pre- and post-repair work. This enables the implementation of necessary countermeasures and tasks. Furthermore, by accumulating and managing unified underground and above-ground four-dimensional information, including location and time information, along with any related information, it is possible to provide a four-dimensional mapping information platform that stores unified underground and above-ground four-dimensional information and related useful information.

[0093] Furthermore, the orthomosaic images obtained in this embodiment have the same accuracy as, for example, a 1 / 500 scale plan (map), while allowing for a visual understanding of the local situation based on the image. Visually superior to maps, they can provide useful, added-value results and information that cannot be obtained from a simple plan (map). Furthermore, in this embodiment of the four-dimensional mapping information system, high-precision three-dimensional coordinate information can be easily obtained from the omnidirectional continuous unified processing of exploration results and images simply by driving the exploration vehicle. This has the excellent effect of enabling rapid responses not only in underground exploration such as finding cavities, but also in emergencies, such as ensuring the safety of emergency transport routes during disasters.

[0094] Thus, according to the ground-penetrating exploration device of one embodiment of the present invention, it is possible to acquire highly accurate and precise data as ground-penetrating exploration data while effectively protecting the three-dimensional ground-penetrating radar 20 which serves as a sensor for ground-penetrating exploration, and to generate more accurate and precise positional information that cannot be obtained by GPS as underground and surface positional information, thereby obtaining highly accurate three-dimensional information in which underground and surface information are unified. Therefore, according to this embodiment, ground-penetrating exploration data and surface information can be generated and displayed as a single, unified three-dimensional information set, unlike conventional techniques which simply display ground-penetrating exploration data and plan views (maps, etc.) side by side. This allows ground-penetrating exploration data indicating cavities, buried pipes, etc., to be displayed integrally within accurate orthophotos showing surface images, enabling identification and understanding. This makes it possible to perform accurate and high-precision ground-penetrating exploration that cannot be obtained with conventional techniques, and to provide information useful for infrastructure development, disaster recovery and reconstruction, and pre-disaster recovery.

[0095] In particular, according to the four-dimensional mapping information generation system of this embodiment, highly accurate three-dimensional underground information and three-dimensional above-ground information are synchronized and unified, and time axis information is added to these unified underground and above-ground three-dimensional information for storage and management. This allows for the extraction of unified underground and above-ground data and real-time images of the ground at different times, for comparison and reference. Even if the ground conditions and landscape change due to a disaster, the familiar, realistic, all-around ground imagery allows users to confirm the location of buildings and other structures. Furthermore, underground information for the same location can be easily viewed and referenced on the same screen.

[0096] For example, during tsunamis or large-scale fires, above-ground structures may be washed away, moved, or destroyed, but buried structures rarely change their relative position even during major earthquakes. Therefore, if three-dimensional images of the underground are acquired in advance, in the event of a disaster, the three-dimensional positional information of buried objects underground can be used as benchmark and mapping information, and then it becomes possible to compare and refer to images of the ground during normal times and during a disaster. Furthermore, because the four-dimensional mapping information generated in this embodiment links points in the image using relative coordinates through CV calculations, it becomes possible to quickly and accurately determine the position even if there is a shift in the absolute coordinates of the reference point during a disaster.

[0097] Therefore, by extracting, comparing, and analyzing four-dimensional mapping information, it becomes possible to implement necessary responses and measures for recovery and reconstruction without having to travel to the site to investigate the detailed extent of the damage, thereby achieving early recovery from the disaster. Furthermore, from the perspective of more thorough pre-disaster recovery, it is desirable to accumulate video data from all over Japan, for example, as the basis for four-dimensional mapping information. In that case, the amount of data would be enormous, and identifying and extracting comparative images of normal times and post-disaster conditions would have limitations if processed solely by human visual inspection. However, for example, AI-based image recognition functions could be used, which would make it possible to reduce manpower and improve efficiency in assessing infrastructure deterioration over time and damage during disasters. [Examples]

[0098] Hereinafter, a more specific embodiment of the four-dimensional mapping information generation system according to the present invention described above will be explained with reference to Figures 23 to 36. The present invention will be described in more detail by the following examples, but the present invention is not limited in any way by the following examples.

[0099] As described above, the present invention relates to ground-penetrating exploration technology that non-destructively and indirectly explores and investigates underground conditions by utilizing the physical properties of the ground to seismic waves, electrical resistivity, electromagnetic waves, ultrasonic waves, etc., as exemplified by ground-penetrating radar exploration. In particular, an embodiment of a position deformation correction device and system using camera images and ground-penetrating radar will be described. Conventional ground-penetrating exploration technology can accurately identify and locate underground cavities, buried pipes, and other structures on the ground surface. However, these technologies assume that absolute coordinates are constant, and when liquefaction or lateral flow occurs due to earthquakes or other events, the location changes, making it difficult to accurately grasp the amount of local displacement before and after the disaster.

[0100] If there are no landmarks to pinpoint the location of the ground surface after a change in elevation, it becomes impossible to correlate camera footage of the ground surface with previously captured data. In particular, when above-ground structures disappear and localized ground deformation such as liquefaction, landslides, and ground fissures occurs, it becomes difficult to compensate for these deformations. As a result, it becomes impossible to determine the amount of localized deformation, leading to problems such as delays in ground improvement and construction planning.

[0101] Furthermore, interferometric SAR analysis using two or more SAR (Synthetic Aperture Radar) images acquired by satellites at different times makes it possible to obtain the spatial distribution of surface deformation relative to the line of sight as seen from the satellite. However, the displacement obtained by this method is the displacement in the direction as seen from the satellite. In order to accurately measure the horizontal displacement, it is necessary to separately determine the vertical displacement using another measurement method and convert it into the surface displacement component. Furthermore, while it is possible to measure the amount of deformation using electronic reference points, there is a problem in that the wide spacing between the electronic reference points makes it impossible to respond to localized ground deformation.

[0102] In deriving fluctuation amounts based on comparisons of measurement results using multi-channel ground-penetrating radar, which is used as a ground-penetrating exploration method, the combinations of data to be compared become extremely large because the ground measurement information is obtained as three-dimensional information in the direction of propagation, channel width direction, and time axis. Furthermore, because the propagation speed of electromagnetic waves differs depending on the water content of the soil, the converted depth obtained by converting time into depth will not be the same, making it difficult to compare two sets of underground measurement data. Furthermore, changes in buried objects due to construction work, or changes in water content, can increase the attenuation of electromagnetic waves, potentially resulting in data that was captured in one measurement being not captured in another. Furthermore, because it is expressed as the amplitude of the reflected wave, it has the same value at many points, making it difficult to directly adopt comparison methods used for images.

[0103] Even if a reflected signal is obtained, the electromagnetic wave propagation speed in the ground is unknown in profile measurement because it is three-dimensional data, and furthermore, the ground surface height may fluctuate due to sediment accumulation. Therefore, the number of possible combinations when comparing the distribution of underground information becomes enormous, making analysis difficult.

[0104] The position change correction device and system according to one embodiment of the present invention shown below are configured and function as a centralized information extraction unit 62 (see Figure 8) of the four-dimensional mapping information generation unit 60 described above, and can correct based on the spatial pattern of the reflected signal of the ground-penetrating radar. This allows for correction of localized movements, offering the advantage of correcting positional changes even under conditions where the measurement path does not match or where the arrangement of ground features has changed.

[0105] A method for generating four-dimensional mapping information using a four-dimensional mapping information generation system according to one embodiment of the present invention, which takes two pieces of ground-penetrating survey measurement information, each with positional information for a first coordinate system and a second coordinate system, and three-dimensional ground mapping information as input, will be described with reference to Figure 23 and subsequent figures. As shown in Figure 23, the four-dimensional mapping information generation system (unified information extraction unit 62) of this embodiment is configured to function as follows: horizontal plane subsurface data conversion unit 2301, model extraction unit 2302, model correspondence analysis unit 2303, first deformation estimation unit 2304, second deformation estimation unit 2305, first coordinate mapping information generation unit 2306, deformation calculation unit 2307, and second coordinate mapping information generation unit 2308.

[0106] Specifically, the coordinate system before the earthquake is designated as the first coordinate system, and the coordinate system after the earthquake, where the positional changes occurred, is designated as the second coordinate system. Here, there are no particular constraints on whether the first coordinate system or the second coordinate system is earlier in time; for example, the first coordinate system may be at a later time and the second coordinate system at an earlier time. Furthermore, if it has been confirmed in advance that similar subsurface information can be obtained at different locations, it is possible to set and adapt a first and second coordinate system based on each location.

[0107] [Horizontal surface underground data conversion unit] The horizontal plane underground data conversion unit 2301 takes the position information of the travel route and the underground exploration measurement information provided by the central processing unit as input, and outputs three-dimensional underground information linked to the coordinates of the horizontal plane based on the analysis of reflected waves. The analysis of reflected waves from ground-penetrating radar can be carried out, for example, according to known techniques described in Non-Patent Document 1 above.

[0108] The ground-penetrating radar survey data is composed of signals on a three-dimensional grid, with the scanning interval in the direction of travel, the channel spacing in the lateral direction, and the sampling time in the depth direction. The position of a grid point is specified by an index in each direction. If the measurement path is not a straight line, the cross-sectional view based on the grid will not match the shape of the actual horizontal plane cross-section. Furthermore, if the measurement paths before and after the variation do not perfectly match, the means for comparing the shape of the signals is unclear, making it difficult for the model extraction unit 2302 to extract the model. Therefore, in order to facilitate the comparison of subsurface information before and after the changes, it is necessary to generate three-dimensional subsurface information linked to horizontal plane coordinates.

[0109] In this embodiment, the ground-penetrating exploration device is configured as a ground-penetrating radar device, and the same configuration was used for both the first and second measurements. However, this method is not limited to the same configuration. The ground-penetrating radar device or system used for the first and second measurements may differ, as long as the ground-penetrating measurement information can be appropriately acquired. Specifically, the combinations of ground-penetrating radar devices used in the first and second tests, including different antenna shapes, frequencies, frequency bands, electromagnetic radiation intensity, angular dependence of electromagnetic radiation intensity, distance between transmitter and receiver, and antenna orientation, are not limited. Furthermore, for ground-penetrating radar systems, impulse methods, variable continuous wave methods (step frequency continuous wave methods, chirp continuous wave methods), and pseudo-noise methods are available. There are no restrictions on the combination of these methods, as long as ground-penetrating information can be acquired appropriately.

[0110] Regarding ground-penetrating radar measurement methods, in addition to the most commonly used profile measurement, techniques such as wide-angle measurement, CMP measurement, and MIMO (Multiple-Input and Multiple-Output) measurement can be used to obtain velocity information for generating ground-penetrating data. These measurement methods can be selected or combined and applied in any way, as long as a model indicating the location of buried objects underground can be generated. Furthermore, the scanning interval in the direction of propagation for ground-penetrating radar measurements is not limited to a fixed interval, as long as it has the resolution necessary to generate the ground-penetrating model described later. The interval can be adjusted according to the required accuracy of the generated model. For example, it is possible to set the scanning interval or channel interval in the direction of propagation to 4 cm in one location and change it to 8 cm in another location.

[0111] Referring to Figure 24, the horizontal plane underground data conversion according to this embodiment will be explained. Specifically, Figure 24(a) is a horizontal cross-sectional view based on a grid, with the horizontal axis representing the direction of travel and the vertical axis representing the antenna width. Because the travel path is curved, the linear shape of the buried object is displayed as a distorted shape that differs from its actual shape. Figure 24(b) is a horizontal cross-sectional view based on a coordinate system on the horizontal plane, where buried objects are displayed as straight lines, making it easy to extract the model from the signal. This method is also applied to ground-penetrating survey measurement information in the other coordinate system, creating ground-penetrating information linked to a coordinate system based on the horizontal plane.

[0112] Furthermore, in ground-penetrating radar surveys where the ground surface is not horizontal, the ground-penetrating radar antenna makes contact at an angle, so the normal direction differs from the vertical. As a result, when measuring on a sloped ground surface, a discrepancy occurs between the position of the underground model projected onto the slope and the position projected onto a virtual horizontal plane. Specifically, because the position of the vertex of the hyperbola of the reflected wave from the buried object is different from directly above the object, a discrepancy arises between the apparent location of the buried object and its actual location. Furthermore, tsunamis and other events can cause sediment to accumulate on top of the ground, potentially altering the subsurface composition before and after the upheaval.

[0113] These points can be addressed by following the known techniques described in Non-Patent Document 1 and other documents mentioned above. Through analysis such as static correction and migration signal processing based on the positional information of the ground surface and reflected waves from different paths obtained by multi-static measurements, it is possible to obtain three-dimensional information of the ground located in the correct position within the ground, even in the case of such ground-penetrating radar surveys. Furthermore, the model extraction unit shown below can more accurately calculate the amount of change as the position of the model relative to the horizontal plane. In Figure 24, three-dimensional ground data from a single run is shown as an example in each coordinate system, but ground exploration measurement information from multiple runs can be combined and processed on a horizontal plane. Specifically, it is possible to generate three-dimensional underground data for the entire lane area.

[0114] [Model Extraction Section] The model extraction unit 2302 receives three-dimensional underground data information generated by the horizontal underground data conversion unit 2301, extracts underground buried objects and pipes manually or by algorithm, and outputs an underground model placed on a horizontal plane based on a coordinate system that retains depth information. Furthermore, the model extraction unit 2302 receives the three-dimensional underground data conversion unit 2301, the correspondence between the coordinate systems of the models in both coordinate systems from the model correspondence analysis unit 2303, and the amount of deformation from the first deformation estimation unit 2304, and outputs an underground model placed in the horizontal plane based on the coordinate system that holds the depth information.

[0115] Specifically, the model extraction unit 2302 combines multiple reflection image cross-sections, existing drawings, CAD data, and other materials related to buried objects to extract buried objects and buried pipes, and adds latitude, longitude, and depth information to create a model of the underground. Herein lies the challenge that analyzing the vast amount of reflected wave signals acquired requires skilled experience and is also time-consuming. To address this challenge, a method has been developed that uses machine learning to quantitatively identify and extract buried objects and pipes from three-dimensional underground data.

[0116] Machine learning is a technique that uses computers to analyze data, learn patterns and rules, extract and classify information. It involves creating identification models using convolutional neural networks (CNNs), and training these models using measurement data with known buried object information. Here, preprocessing is performed on the subsurface 3D data, including setting the ground surface, filtering to reduce noise, adjusting the range gain, and adjusting the contrast. Furthermore, migration signal processing can also be applied.

[0117] Specifically, it can learn the intensity, polarity, isolation, shape, and even complex patterns of ground-penetrating radar waveforms to identify cavities, buried objects, and buried pipes. Furthermore, to improve identification accuracy, the amount of training data can be increased using data augmentation. Then, the system can input ground-penetrating radar reflection images to be used for classification into a pre-trained classification model, and output the classified results.

[0118] Subsequently, latitude, longitude, and depth information can be assigned to the locations of the extracted buried objects, and a subsurface model can be created that retains depth information and is placed in a coordinate system. For example, you can create an underground model by enclosing cavities and buried objects with polylines and connecting buried pipes with polylines. Furthermore, it is possible to create a model by projecting the underground model onto the ground surface and display it on a 360-degree video and orthomosaic image.

[0119] Based on the above, the model extraction unit 2302 extracts point, polyline, and polygon models by analysis using horizontal cross-sectional views at multiple depths of the subsurface three-dimensional information. The points that make up the model are composed of (x,y,z) pairs in the first coordinate system and (u,v,w) pairs in the second coordinate system. Here, if the approximate displacement amounts of the first and second coordinate systems are known, then, in order to reduce time, prior information using the model information of the other coordinate systems and the displacement amounts given by the first displacement estimation unit 2304 can be considered once the model extraction in the first coordinate system is completed. Specifically, by analyzing the corresponding region in the second coordinate system where a model exists in the first coordinate system, and excluding the corresponding region in the second coordinate system where a model does not exist in the first coordinate system, it is possible to extract the model in the second coordinate system. Similarly, the extraction of the corresponding model in the second coordinate system within the first coordinate system can be performed.

[0120] [Model-Based Analysis Department] The model-corresponding analysis unit 2303 receives multiple models from different coordinate systems in the model extraction unit 2302, performs an analysis that takes into account changes in the presence or absence of buried objects or pipes, and estimates based on projection transformations and distances that consider the positional relationships of the multiple models. It outputs the correspondence between the models in both coordinate systems.

[0121] [First fluctuation estimation unit] The first variation estimation unit 2304 takes multiple models before and after the variation and their corresponding relationships as input, estimates the global variation, and outputs it. In this embodiment, the first variation estimation unit 2304 performs alignment based on the model correspondence relationship provided by the model correspondence analysis unit 2303.

[0122] Specifically, the positional relationship of the underground in the two coordinate systems can be represented by a projection transformation, and the first deformation estimation unit 2304 can estimate the parameters of this transformation. For example, if the presence or absence of buried objects or pipes changes due to construction or other reasons, outliers can be excluded using the robust estimation method RANSAC (RANdom SAmple Consensus). Based on the correspondence between multiple models, the projection transformation parameters can be estimated for the combination that minimizes the difference between the position given by the projection transformation and the position of the model in the second coordinate system.

[0123] The transformation between the two coordinate systems is a projection transformation, and the parameter is represented by x. The derivation of homography matrices based on the correspondence between points, lines, and curves is described, for example, in Non-Patent Document 2 mentioned above.

[0124] Specifically, the correspondence between a point (x,y) in the first coordinate system model and a point (u,v) in the second coordinate system model is expressed as (x,y,1) and (u,v,1) using homogeneous coordinates. The projective transformation for this is expressed by the homography matrix in the following equation 1. hij represents the element in the i-th row and j-th column of the homography matrix. The h33 of the homography matrix is ​​1 and has 8 degrees of freedom. [Math 1] TIFF2026074839000002.tif2072

[0125] For affine transformations with more constraints, the following equation 2 applies. There are 6 degrees of freedom, where aij is the element in the i-th row and j-th column of the affine matrix, and t1 and t2 are values ​​corresponding to translations in the x and y directions, respectively. [Formula 2] TIFF2026074839000003.tif2068

[0126] Furthermore, in the case of coordinate transformations involving constrained rotation and translation, it can be expressed by the following equation 3. θ is a value corresponding to the angle of rotational transformation with respect to an axis perpendicular to the plane. This transformation is represented by three degrees of freedom. [Formula 3] TIFF2026074839000004.tif2079

[0127] Given the correspondence between multiple points in a model across two coordinate systems, the problem becomes solving for the homography matrix of the projection transformation, which can be solved using the DLT (Direct Linear Transformation) algorithm. The mathematical procedure is publicly known and is shown, for example, in Non-Patent Document 2 mentioned above.

[0128] However, in the case of subsurface three-dimensional information, if, for example, the presence or absence of buried objects changes due to construction work, or if reflection cannot be detected due to water content, it may be impossible to extract some models that match one of the coordinate systems. Even in such cases, the model correspondence and homography matrix can be derived using a known technique called MSAC (M-estimator Sample Consensus), an improved version of the RANSAC algorithm, which determines the combination of models and finds the optimal parameters.

[0129] As a means of deriving a more reliable model correspondence and homography matrix, it is conceivable to apply iterative processing to the model extraction unit 2302. First, the homography matrix and variation are calculated for the combinations obtained by the model-compatible analysis unit 2303. Next, the model extraction unit 2302 is provided with the correspondence relationship between the models in both coordinate systems from the model correspondence analysis unit 2303 and the amount of variation from the first variation estimation unit 2304, and this prior information is given to the model extraction unit 2302. This improves the accuracy of model detection by performing a detailed analysis of model extraction. If no new models are extracted as a result of this process, or if the difference compared to the previous variation falls within the specified threshold, the iteration of the procedure is terminated.

[0130] Figure 25 is a schematic diagram illustrating a method for extracting and modeling features based on two sets of three-dimensional underground data and estimating the amount of deformation. As shown in the figure, if there are buried objects underground, a reflected signal will be present, allowing the area in question to be extracted.

[0131] The black area in Figure 25(a) is a schematic diagram showing the region where buried objects exist on the two-dimensional XY plane, extracted from three-dimensional underground information based on the first coordinate system (X,Y,Z). The grid lines indicate directions along the coordinate system. In Figure 25(b), the center of the circular shape is assigned as a point model based on its circular characteristics. Figure 25(c) is a schematic diagram showing the region where buried objects exist in the two-dimensional UV plane, extracted from three-dimensional subsurface information based on the second coordinate system (U,V,W). Similar to Figure 25(b), in Figure 25(c) the center of the circular feature is assigned as a point model.

[0132] As shown in these figures, the transformation matrices given by translation and rotation can be derived based on the coordinates of three points in the first and second coordinate systems. In coordinate transformations, since the amount of variation varies depending on the location, it is possible to divide the spatial region and apply the transformation accordingly, or to divide the depth into multiple intervals and treat each interval as having its own amount of variation.

[0133] Figure 26 is a schematic diagram illustrating a method for estimating the amount of variation when a model of three straight lines exists on the planes of the first and second coordinate systems. As shown in the figure, the first coordinate system corresponds to the second coordinate system through translation and rotation. The two points located at the ends of each line characterize the position and direction of that line. From these two points, we can derive an equation representing the line, and from the relationship between the corresponding line models, we can obtain the homography matrix of the projective transformation.

[0134] Figure 26(a) shows a model of buried pipes L1, L2, and L3 in the XY plane of the first coordinate system. The line L1 is determined by passing through two points P1,1 and P1,2. Lines L2 and L3 are determined in the same manner. Figure 26(b) shows a model of the lines L1, L2, and L3 on the UV plane of the second coordinate system. The line L1 is determined by passing through two points Q1,1 and Q1,2. The equations for other lines can be derived using a similar procedure.

[0135] The gray rectangle in Figure 26(b) represents the exploration area in the first coordinate system after projection transformation. Because it differs from the exploration area in the second coordinate system, the positions of the ends of the linear model are also different, and P1,1 does not correspond to Q1,1 after projection transformation. Therefore, it is necessary to treat this not as a problem of corresponding points between two coordinate systems, but as a problem of deriving the homography matrix of the projection transformation based on the equations of the linear model. The method for solving this problem is publicly known, as shown in Non-Patent Document 2 mentioned above.

[0136] Figure 27 is a schematic diagram illustrating a method for estimating fluctuations when a model exists that assumes buried pipes with complex shapes such as L-shapes and T-shapes. The figure shows a model of a buried pipe with three bends that lie on the planes of the first and second coordinate systems. Specifically, Figure 27(a) shows the model on the XY plane of the first coordinate system, and Figure 27(b) shows the model on the UV plane of the second coordinate system, with the endpoints, intersections, and transition points of the continuous linear shape extracted as characteristic elements of the model. Furthermore, Figure 27(b), similar to Figure 26(b) described above, shows the exploration area in the first coordinate system, projected using gray rectangles.

[0137] As shown in these figures, in the exploration area of ​​the second coordinate system, model L1, which has a bent section, is displayed separately in two locations. Even in this case, the model of the first coordinate system can be represented as multiple equations specified by feature points of a continuous line shape, and even for models with complex shapes, the homography matrix of the projection transformation can be derived.

[0138] Figure 28 is a schematic diagram illustrating a method for estimating fluctuations when a model exists that assumes three curved buried pipes on the planes of the first and second coordinate systems. As shown in the figure, to represent the curve shape of the model, points at specified distances can be extracted as elements that characterize the curve, or points where the inflection point or slope is a specific value such as horizontal or vertical, or intersection points with other curves can be specified, or a more general curve can be represented as a NURBS (Non-Uniform Rational B-Splines) curve.

[0139] Figure 28(a) shows the model on the XY plane of the first coordinate system, and Figure 28(b) shows the model on the UV plane of the second coordinate system. In Figure 28(b), similar to Figure 26(b) described above, the exploration area in the first coordinate system, transformed by projection, is shown as a gray rectangle. Even in this case, the model of the first coordinate system can be expressed as multiple equations, and even for models with complex shapes, the homography matrix of the projection transformation can be derived.

[0140] [Second Variation Estimation Unit] The second deformation estimation unit 2305 takes as input the deformation amount calculated by the first deformation estimation unit 2304 and the three-dimensional underground information linked to the horizontal plane coordinates generated by the horizontal plane underground data conversion unit 2301, and divides the area into multiple regions based on the measurement path. Then, to make the subsurface three-dimensional information for the first and second coordinate systems similar and to facilitate subsequent comparison of local features, range gain adjustment, contrast adjustment, and derivation of the correspondence between depth and subsurface three-dimensional information are performed for each region. Furthermore, pre-processing such as migration signal processing and noise reduction is applied to generate pre-processed subsurface three-dimensional information.

[0141] As a result, the second variation estimation unit 2305 can generate multiple planar cross-sectional views of different depths and find matching locations for features representing local sub-regions in the image for two of the cross-sectional views. Furthermore, the second deformation estimation unit 2305 can also identify locations with matching features using three-dimensional local features from pre-processed three-dimensional underground information as an alternative method. Furthermore, the second variation estimation unit 2305 uses the variation amount calculated by the first variation estimation unit 2304 to perform outlier filtering based on a specified distance or the like for locations where local features obtained at multiple depths coincide. Then, based on the variation in the filtered feature matching areas, statistical processing such as outlier removal and averaging can be performed to calculate and output the variation at the center of each region.

[0142] Figure 29 is an explanatory diagram showing the path-based region division method in the second variation estimation unit 2305. The diagram shows paths based on curves formed by the edges of subsurface data on a latitude and longitude plane. Specifically, for example, a single area can be defined as 25m x 25m and divided into a 400x400 grid, and three-dimensional underground information for each of these grids can be obtained through interpolation. After that, specify the distance along the exploration path and specify the center of the next divided region. This method generates two sets of three-dimensional underground data based on a first and second coordinate system, performs preprocessing, and generates cross-sectional diagrams for comparison between the respective regions. Note that the dimensions and shapes of the domain divisions in this method are not limited to those described above.

[0143] Here, regarding three-dimensional underground information in two coordinate systems within a specified area, the electrical properties and soil composition may change due to factors such as changes in soil water content or sediment deposition. If the electrical properties of the medium differ, the propagation speed and absorption coefficient of electromagnetic waves will change, resulting in a change in the intensity of reflected waves from buried objects. Therefore, in this embodiment, in order to facilitate comparison based on cross-sectional images of three-dimensional underground information, preprocessing is performed to adjust the range gain of reflection intensity, adjust the contrast, and correct the converted depth, and a horizontal cross-sectional image for comparison can be generated based on the corrected underground information. Specifically, the signal gain is adjusted and the correspondence between the Z and W depth directions in the two coordinate systems is adjusted based on the distribution of reflected wave intensity at each depth within the region in the first and second coordinate systems. Furthermore, migration signal processing and filtering can be performed to remove noise.

[0144] In cross-sectional diagrams generated based on ground-penetrating information corrected through such preprocessing, it is possible to combine data measured using different radar methods or to use real and imaginary part information obtained using the step frequency continuous wave method. Furthermore, it is possible to use multiple measurement data from different frequency bands, multiple data from measurements with different polarizations, and measurement data from spatial arrangements of antenna configurations. For example, it is possible to use data with different distances and orientations between the transmitter and receiver, or to generate cross-sectional images of these numerous channels.

[0145] Furthermore, the second variation estimation unit 2305 can determine the amount of variation relative to the center of each region. Since ground-penetrating radar reflection signals have the same value at many locations, it is desirable to perform distance-based filtering to eliminate correspondences with significantly different fluctuations. Therefore, in this embodiment, in the filtering process of the second variation estimation unit 2305, a threshold value for distance can be set using the variation amount calculated by the first variation estimation unit 2304 or the variation amount based on the homography matrix representing the projection transformation. Specifically, assuming that the survey areas of both overlap, in addition to the fluctuations of the first fluctuation estimation unit 2304, for example, for 5m, which is more than approximately twice the antenna width of 2.4m, no change in the antenna width direction can occur, and therefore it can be set as an explicit value. Furthermore, it is possible to set thresholds for fluctuations in neighboring areas, such as a 10% fluctuation or 5m.

[0146] Figure 30 is a cross-sectional view of the ground-penetrating radar reflection image in the region from the second variation estimation unit 2305. In the figure, the upper part shows underground information for the first coordinate system, and the lower part shows underground information for the second coordinate system. Furthermore, in the same figure, the left side shows a horizontal cross-section, the center shows a cross-section in the Y direction relative to the crosshair, and the right side shows a cross-section in the X direction relative to the crosshair. Based on these diagrams, it becomes possible to read the corresponding reflected signals from the drawings and specify them manually. Furthermore, based on the amount of variation calculated by the first variation estimation unit 2304, it is possible to display the position of the crosshair in the first coordinate system on the lower side in correspondence with the position of the crosshair in the second coordinate system after projection transformation.

[0147] As mentioned above, the propagation speed of electromagnetic waves used in ground-penetrating radar surveys depends on the electrical properties of the medium, such as the relative permittivity, and changes depending on the water content and soil type in the ground. Furthermore, the composition changes due to sediment deposition, etc. In these cases, it is necessary to determine the correspondence between the depths of the two coordinate systems. The depth directions of the first and second coordinate systems are Z and W, respectively. Figure 31 is a cross-sectional view with the horizontal axis representing the horizontal direction and the vertical axis representing the depth direction, illustrating how the correspondence between the depths of two coordinate systems is derived using a typical reflected signal.

[0148] Specifically, Figure 31(a) is a cross-sectional view of the first coordinate system, and Figure 31(b) is a cross-sectional view of the second coordinate system. In these diagrams, the location of the buried object, indicated by the black circle, corresponds to depths z1 as w1 and z2 as w2. By interpolating for other sections, the correspondence between depths in the two coordinate systems can be determined. Furthermore, it is possible to determine the correspondence between depths in two coordinate systems using not only buried objects but also geological boundaries, etc.

[0149] Here, AKAZE (Accelerated-KAZE) is known as a method that is less susceptible to noise and can efficiently extract consistent feature points even with geometric deformations. For example, Non-Patent Document 3 described above describes AKAZE, one of the methods for extracting matching local feature points between two images. Furthermore, as described in Non-Patent Document 4 above, it is known that local feature matching methods utilizing deep learning such as LightGlue can be used to extract matching local features and derive displacement amounts. Therefore, these known methods can also be employed in this embodiment.

[0150] Figure 32 shows the correspondence between local features extracted by AKAZE and horizontal cross-sectional views generated from two coordinate systems. In the figure, the left side is a horizontal cross-sectional view of the ground-penetrating radar in the first coordinate system, and the right side is a horizontal cross-sectional view of the ground-penetrating radar in the second coordinate system. The endpoints of the line connecting the left and right images indicate the locations where the local features of both images are estimated to coincide.

[0151] Figure 33 shows, on a plane, the coordinates and their respective fluctuation amounts for which local features matched on horizontal cross-sectional views of two coordinates using AKAZE at multiple depths within a divided region. In the figure, outliers of the variation calculated using the RANSAC algorithm are shown as black arrows, and the elements used to derive the variation in this region are shown in gray. In this way, outliers can be removed based on the obtained variation.

[0152] Alternatively, one can use three-dimensional local features to extract matching locations from pre-processed subsurface three-dimensional information based on two coordinate systems. Figure 34 shows a model of the three-dimensional shape of a cavity extracted from three-dimensional underground data. Examples of such known three-dimensional local features are shown in Non-Patent Documents 5 and 6 mentioned above. However, the methods for extracting feature points are not limited to these.

[0153] Furthermore, if there is a two-layer structure of variation in depth, it can be analyzed by considering it as multiple depth intervals. In this case, there are multiple coordinate transformations for a given point on the horizontal plane, and multiple variation amounts are given. Figure 35 is a schematic diagram illustrating a situation where there are two different levels of variation depending on the depth. In the figure, the horizontal direction is the horizontal axis and the depth direction is the vertical axis. Figure 35(a) is a cross-sectional view of the first coordinate system, and Figure 35(b) is a cross-sectional view of the second coordinate system. As shown in these figures, for example, since the amount of displacement of the two buried pipes in the upper layer and the amount of displacement of the two buried pipes in the lower layer are different, it can be analyzed as having two amounts of displacement corresponding to each depth.

[0154] [First Coordinate Mapping Information Generation Unit] The first coordinate mapping information generation unit 2306 generates known information and surrounding feature information corresponding to the first coordinate system. Specifically, using the 360-degree camera images mentioned above (see Figure 6), it is possible to generate mapping information that uses existing information such as the location of ground features and local government maps as location elements (see Figure 10).

[0155] [Variable Calculation Unit] The deformation calculation unit 2307 takes the deformation amount at representative positions of each divided region from the second deformation amount estimation unit 2305 and the feature information from the first coordinate mapping information generation unit 2306 as input, provides the deformation amount in the first and second coordinate systems at an arbitrary position, and outputs the position information of the feature corresponding to the second coordinate. Furthermore, the inverse transformation from the second coordinate system to the first coordinate system is also possible.

[0156] [Second Coordinate Mapping Information Generation Unit] The second coordinate mapping information generation unit 2308 generates mapping information for features based on the position in the second coordinate system for each element of the mapping information of the first coordinate calculated by the variation calculation unit 2307. For camera images captured by an all-around camera, etc., a pseudo-CV value can be calculated for another coordinate system using the variation calculation unit 2307. This allows information about the vicinity of the other path to be displayed based on the converted measurement path. However, if pseudo-CV values ​​are used, distortion may occur in the measurement results, so caution is required when obtaining accurate distances and measurements.

[0157] Furthermore, the second coordinate mapping information generation unit 2308 can create textures for the polygons to be displayed from images obtained through camera analysis and display them on the mapping information of another coordinate system. Furthermore, based on brightness, contrast, camera position and orientation, and time information in the surrounding images, it is possible to generate shadows that take into account the effects of weather and sunlight, and to create composite images that blend seamlessly with the surroundings using deep learning known as image harmony, which can then be displayed and used as mapping information.

[0158] Figure 36 is a schematic diagram illustrating the method for generating mapping information for a second coordinate system, which integrates ground and underground information, in the second coordinate mapping information generation unit 2308. Specifically, Figure 36(a) shows pre-provided road information, and based on multiple images with CV values ​​created from 360-degree camera footage, information such as the edge of the road shoulder, the position of white lines, manholes, and the location of house boundaries is generated. Figure 36(b) is a schematic diagram that combines road information with surrounding feature information (such as land boundaries and building shapes). This allows for detailed information about the area around the road. Specifically, it can incorporate information such as maps from the Geospatial Information Authority of Japan and 3D information about buildings.

[0159] Figure 36(c) shows the result of converting mapping information, which consists of points, line segments, polylines, and polygons, from the first coordinate system to the second coordinate system based on the amount of variation obtained in this embodiment, and displaying it superimposed on the second coordinate system. FIG. 36(d) is a diagram schematically showing a case where mapping information of the first coordinate system converted into the second coordinate system is displayed on a camera image. In this case, although not particularly shown, it is also possible to display, with texture, a model created from the other omnidirectional camera images for surrounding ground features.

[0160] As described above, the preferred embodiments and examples of the four-dimensional mapping information generation system of the present invention have been shown and described. However, the present invention is not limited to the above-described embodiments and examples, and it is needless to say that various modifications can be made within the scope of the present invention. For example, in the above-described embodiment, as the underground exploration means (sensor) in the four-dimensional mapping information generation system according to the present invention, a radar using electromagnetic waves (three-dimensional underground radar 20) has been taken as an example for explanation. However, the underground exploration means is not limited to a radar using electromagnetic waves as long as it can acquire and generate three-dimensional information of the underground. Specifically, as the underground exploration means, in addition to the case of using electromagnetic waves, sensors using seismic waves, electrical resistivity, ultrasonic waves, radar, etc. may be used.

[0161] Also, in the above-described embodiment, the ground image generated and output as four-dimensional mapping information has been described by taking a real image captured and acquired by an in-vehicle camera (omnidirectional camera 30) of a survey vehicle as an example. However, as an image or picture showing the ground, instead of the real image or together with the real image, for example, a 3D image or point cloud data can also be displayed.

Industrial Applicability

[0162] The present invention can be suitably used for underground exploration technologies such as ground penetrating radar exploration, which indirectly explores and surveys the underground (subsurface) situation nondestructively by utilizing the physical properties of the ground with respect to seismic waves, electrical resistivity, electromagnetic waves, ultrasonic waves, etc.

Explanation of Reference Numerals

[0163] 10 Survey vehicle 11 GPS (High-precision GNSS) 20. Three-dimensional ground-penetrating radar (ground-penetrating exploration means) 21 Radar Unit 22 Antennas 30. Omnidirectional camera (ground image generation means) 40 CV (Camera Vector) Calculation Unit 50. Unified Processing Unit (Unified Processing Means) 60 Four-dimensional mapping information generation unit 61. Centralized Information Storage Unit 62 Centralized information extraction unit 63. Comparison Image Generation Unit

Claims

1. A ground-penetrating exploration means is installed on a road-traveling exploration vehicle and generates three-dimensional underground information beneath the road surface on which the exploration vehicle travels. The exploration vehicle is equipped with a ground image generation means that generates a three-dimensional ground image of the road surface on which the exploration vehicle travels, A data unification processing means generates unified information from underground to surface space by performing a predetermined data unification process on the three-dimensional underground information generated by the underground exploration means and the three-dimensional surface image generated by the surface image generation means. A unified information storage means that stores multiple unified pieces of information generated at different times, with time axis information indicating the generation time added to them. The system includes a unified information extraction means that extracts multiple unified pieces of information from the accumulated unified pieces of information, based on the underground three-dimensional information, that are generated at different times but have the same above-ground location, This generates four-dimensional mapping information by adding time-axis information to unified three-dimensional information of underground and above-ground data. A four-dimensional mapping information generation system characterized by the following features.

2. The system includes a comparison video generation means that outputs multiple ground-based three-dimensional images, each generated at a different time but with the same ground position, from the extracted multiple unified pieces of information, in a comparative format. The four-dimensional mapping information generation system according to feature 1.

3. The unified information extraction means is, Based on the three-dimensional positional information of underground cavities and / or buried objects, multiple unified pieces of information are extracted that are generated at different times but have the same above-ground location. A four-dimensional mapping information generation system according to claim 1 or 2, characterized in that it is the same as described in claim 1 or 2.

4. The aforementioned unified information storage means is As multiple unified pieces of information with different generation times but matching ground locations, It stores unified information generated during peacetime and unified information generated during disasters. A four-dimensional mapping information generation system according to claim 1 or 2, characterized in that it is the same as described in claim 1 or 2.

5. The aforementioned terrestrial image generation means A feature point extraction unit automatically extracts a predetermined number of feature points from the image data of the aforementioned three-dimensional terrestrial video, A feature point matching processing unit automatically tracks the extracted feature points within each frame image of the video footage to determine the correspondence between the frame images, The system includes a camera vector calculation unit that calculates the three-dimensional position coordinates of the feature points for which a correspondence has been determined, and from these three-dimensional position coordinates, calculates a camera vector consisting of the three-dimensional position coordinates and three-dimensional rotation coordinates of the camera corresponding to each frame image, and has a camera vector calculation means. The aforementioned unified processing means, Based on the camera vector generated by the camera vector calculation means and the position coordinate information contained in the underground three-dimensional information, the underground three-dimensional information and the above-ground three-dimensional image are unified. A four-dimensional mapping information generation system according to claim 1 or 2, characterized in that it is the same as described in claim 1 or 2.

Images