Travelling possibility display system, travelling possibility display method, travelling possibility display control device, and travelling possibility display control program

Abstract

To allow workers to determine at a glance whether a vehicle can travel at an actual construction site.SOLUTION: A travelling possibility display system 1 displays information about the travelling possibility of a vehicle on a road on which the vehicle is scheduled to travel, and includes a road condition acquisition device 3 that acquires point cloud data of the road and surrounding objects along the road, a travelling possibility prediction device 4 that uses the point cloud data to recreate the road and the roadside in a virtual space and predicts a travelling possibility by simulating the vehicle's travel in the virtual space, and a display device 5 that displays a fused image in which a travelling possibility prediction result is fused with an image obtained by photographing the road and the roadside. The prediction result includes at least either border crossing range information that indicates a range in which the vehicle will cross the road, or interference range information that indicates a range in which the vehicle will interfere.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a drivability display system, a drivability display method, a drivability display control device, and a drivability display control program. [Background technology]

[0002] Wind energy is attracting attention because it can be converted into electrical energy with high efficiency and, unlike solar power generation, can generate electricity even at night as long as there is wind. In onshore wind power generation facilities, the height of the tower that makes up the power-generating wind turbine (a tower-shaped structure) can be, for example, 50 meters or more. Power-generating wind turbines are divided into multiple parts and transported to the installation site, but these multiple parts are also generally long. For this reason, when transporting the turbines to the installation site using a transport vehicle such as a trailer, it is extremely important to determine in advance whether the transport vehicle can dwell.

[0003] In particular, onshore wind power generation facilities are often built in mountainous regions. In such cases, transport vehicles loaded with winding roads (mountain roads) in the mountainous areas must travel. If the curvature or width of the road is insufficient for the transport vehicles to travel, advance preparation work such as widening the road is required. Furthermore, if the wind turbine parts mounted on the loading platform interfere with objects along the road (earth and sand, trees, utility poles, road signs, etc.), the objects must be removed.

[0004] Conventional preparatory work involves, for example, the following steps (1) to (5): (1) On-site reconnaissance and surveying. (2) Examining the vehicle's travel path on a road alignment map (2D plan view) obtained from a map or satellite image. (3) Measuring the location of potential obstacles on-site. (4) Determining whether or not there is potential for interference. (5) Checking the 2D plan view for obstacles that are determined to be potential interference and creating an obstacle removal plan. Checking areas that require road widening and creating a road widening plan. Calculating the labor required for removal and widening. Construction work is then carried out. Steps (1) and (2) are typically performed during the basic design stage by a contractor who creates a road plan, while steps (3) to (5) are typically performed during the detailed design stage or by a contractor who actually performs the removal and widening work. This poses challenges, including the need to visit the site multiple times, resulting in duplicated work.

[0005] In relation to this, technologies for determining the drivability of a vehicle have been developed in the past (see, for example, Patent Document 1). Patent Document 1 describes a driving determination device that determines the drivability of special vehicles and the like. This driving determination device generates a three-dimensional driving trajectory of a vehicle on a road designated as a target for driving determination, assuming that the road is a horizontal plane. The driving determination device also acquires three-dimensional point cloud data collected about the road surface of the designated road and objects in the surrounding space. The driving determination device also calculates the gradient and undulation of the road surface based on the point cloud data of the road surface of the designated road, and corrects the height and inclination of the three-dimensional driving trajectory. The driving determination device superimposes the corrected three-dimensional driving trajectory on three-dimensional point cloud data of objects in the surrounding space of the road, and assigns collision information to attributes of points included in the corrected three-dimensional driving trajectory among the points in the three-dimensional point cloud data of objects in the surrounding space of the road. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 2017-167936 Summary of the Invention [Problem to be solved by the invention]

[0007] Conventional technology involves running a vehicle through a virtual space created on a computer to determine whether or not there is a collision, and then displaying the results on a screen. This makes it difficult for construction companies performing preparatory work such as removal and widening to determine at a glance which objects at the actual construction site (real world) will collide. This means that it is necessary to correlate the results of the simulation in virtual space with the real world, which is time-consuming for construction companies.

[0008] From this perspective, the present invention provides a drivability display system, a drivability display method, a drivability display control device, and a drivability display control program that allow for a glance at the drivability of a vehicle at an actual construction site. [Means for solving the problem]

[0009] The drivability display system according to the present invention is a system for displaying information relating to the drivability of a vehicle on a road on which the vehicle is to travel. The drivability display system includes a road condition acquisition device, a drivability prediction device, and a display device. The road condition acquisition device acquires point cloud data of the road and roadside. The drivability prediction device uses the point cloud data to recreate the road and the roadside in a virtual space, and predicts drivability by performing a driving simulation of the vehicle in the virtual space. The display device displays a fused image in which the predicted result of drivability is fused with an image of the road and the roadside. The prediction result includes at least one of border crossing range information indicating the range in which the vehicle crosses the road and interference range information indicating the range in which the vehicle interferes.

[0010] When acquiring the point cloud data, the road condition acquisition device acquires coordinates of the road and the roadside. The travel possibility prediction device outputs the prediction result that reflects the coordinates. The display device acquires the image capturing position when capturing an image of the road and the roadside, and creates the fusion image based on the coordinates of the prediction result and the image capturing position.

[0011] The travelability prediction device includes, for example, a model creation unit, a travel verification unit, and a travelability prediction unit. The model creation unit reproduces the road and the roadside in a virtual space using point cloud data of the road and the roadside. The driving verification unit performs a driving simulation of the vehicle in the virtual space. The travelability prediction unit predicts the travelability by determining at least one of the range in which the vehicle crosses the road and the range in which there is interference with the vehicle.

[0012] The drivability display system of the present invention displays a fusion image that associates the results of a simulation in virtual space with the real world, allowing users to determine at a glance whether a vehicle can drivably at an actual construction site.

[0013] The prediction result may further include a vehicle model that reproduces the vehicle, and trajectory information that indicates a trajectory of a specific portion of the vehicle. This makes it easier to visualize the vehicle running, and makes it easier to determine whether the vehicle is drivable.

[0014] If the vehicle has a load, the prediction result may further include a vehicle model that reproduces the vehicle, a load model that reproduces the load, and trajectory information that shows the trajectory of a specific location of the load. This makes it easier to visualize the vehicle traveling and the range affected by the load, making it easier to determine whether the vehicle is drivable.

[0015] The display device may display a transparency adjustment section together with the fusion image, and change the transparency of the prediction result in response to an operation of the transparency adjustment section and display the prediction result. This allows the results of the simulation in the virtual space to be compared with the real world, making it easier to determine whether the vehicle is drivable.

[0016] The drivability display method according to the present invention is a method for displaying information relating to the drivability of a vehicle on a road on which the vehicle is to travel. The drivability display method includes a road condition acquisition step, a drivability prediction step, and a display step. In the road condition acquisition step, point cloud data of the road and roadside is acquired. In the drivability prediction step, the road and the roadside are reproduced in a virtual space using the point cloud data, and a driving simulation of the vehicle is performed in the virtual space to predict the drivability. In the display step, a fusion image is displayed in which the predicted result of drivability is fused with an image of the road and the roadside. The prediction result includes at least one of border crossing range information indicating the range in which the vehicle crosses the road and interference range information indicating the range in which the vehicle interferes.

[0017] The drivability display method according to the present invention displays a fusion image that associates the results of a simulation in virtual space with the real world, allowing the user to determine at a glance whether a vehicle can drivably at an actual construction site.

[0018] The drivability display control device according to the present invention is a device that displays information on the drivability of a vehicle on a road on which a vehicle is scheduled to travel on a display unit. The drivability display control device includes a storage unit, a fusion image creation unit, and a display control unit. The storage unit stores the predicted results of the drivability obtained by performing a driving simulation of the vehicle in a virtual space. The fusion image creation unit creates a fusion image by fusing the prediction result with an image of the road and roadside. The display control unit causes the display unit to display the fusion image. The prediction result includes at least one of border crossing range information indicating the range in which the vehicle crosses the road and interference range information indicating the range in which the vehicle interferes.

[0019] The drivability display control device according to the present invention displays a fusion image that associates the results of a simulation in virtual space with the real world, allowing the user to determine at a glance whether a vehicle can drivably at an actual construction site.

[0020] The drivability display control program according to the present invention is a program for displaying information on the drivability of a vehicle on a road on which the vehicle is to be driven on a display unit. The drivability display control program causes a computer having a storage unit that stores a predicted result of the drivability obtained by performing a driving simulation of the vehicle in a virtual space to function as a fusion image creation unit and a display control unit. The fusion image creation unit creates a fusion image by fusing the prediction result with an image of the road and roadside. The display control unit causes the display unit to display the fusion image. The prediction result includes at least one of border crossing range information indicating the range in which the vehicle crosses the road and interference range information indicating the range in which the vehicle interferes.

[0021] The drivability display control program according to the present invention displays a fusion image that associates the results of a simulation in virtual space with the real world, allowing users to determine at a glance whether a vehicle can drivably at an actual construction site. [Effects of the Invention]

[0022] According to the present invention, it is possible to determine at a glance whether a vehicle can be driven at an actual construction site. [Brief explanation of the drawings]

[0023] [Figure 1]1 is a schematic configuration diagram of a travelability display system according to an embodiment of the present invention. [Figure 2] 1 is a diagram illustrating an example of a hardware configuration of a travelability prediction device according to an embodiment of the present invention. [Figure 3] 1 is a diagram illustrating an example of a functional configuration of a travelability prediction device according to an embodiment of the present invention. [Figure 4] FIG. 2 is an image diagram of a vehicle model and a load model. [Figure 5] FIG. 1 is an image diagram of a road environment model. [Figure 6] FIG. 10 is an image diagram of a driving simulation. [Figure 7] FIG. 1 is a diagram illustrating an example of a functional configuration of a display device according to an embodiment of the present invention. [Figure 8] This is the display screen of the fused image when the results of the driving simulation are set to "OFF (non-display)." [Figure 9] This is the display screen of the fused image when the driving simulation results are set to "ON (display)." [Figure 10] 1 is a schematic diagram showing the flow of a travelability display method according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0024] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Each drawing is merely a schematic illustration to allow a sufficient understanding of the present invention. Therefore, the present invention is not limited to the illustrated examples. In each drawing, common or similar components are designated by the same reference numerals, and redundant explanations thereof may be omitted.

[0025] <Configuration of the driving possibility display system according to the embodiment> The configuration of a drivability display system 1 according to an embodiment will be described with reference to FIG. 1. FIG. 1 is a schematic diagram of the drivability display system 1 according to an embodiment. The drivability display system 1 predicts the drivability of a vehicle through simulation, and displays the results of the driving simulation (including the predicted results of drivability) by fusing them with real-world images and videos using augmented reality (AR) technology. For example, the drivability display system 1 uses a terminal carried by a contractor to capture an image of an actual construction site for preparatory work, and displays the results of the driving simulation by overlaying them on the captured image.

[0026] Drivability is a concept that includes at least one of (A) whether a vehicle can travel along a road without going off the road, and (B) whether a vehicle can travel along a road without coming into contact with the road surface, the ground along the road, or objects along the road. In this embodiment, both (A) and (B) above are simulated as drivability, and the results are displayed using augmented reality technology. Note that the drivability display system 1 can also simulate only one of (A) or (B) above and display the results using augmented reality technology. When a vehicle is carrying cargo, the drivability display system 1 may predict drivability including the cargo and display the results using augmented reality technology.

[0027] As shown in Fig. 1, the drivability display system 1 mainly includes a road condition acquisition device 3, a drivability prediction device 4, and a display device 5. The road condition acquisition device 3, the drivability prediction device 4, and the display device 5 are communicably connected via a network. The network may be either wired or wireless.

[0028] The road condition acquisition device 3 acquires point cloud data of roads and surrounding roadside objects. The road condition acquisition device 3 is, for example, a mobile mapping system (MMS). A mobile mapping system is a surveying vehicle 2 equipped with devices for acquiring position and attitude information, and devices for acquiring information on the shape and conditions of the road and its surroundings. By using a mobile mapping system, it is possible to reduce on-site reconnaissance and surveying, and to efficiently acquire three-dimensional point cloud data on the road surface of the target road and roadside objects while the surveying vehicle 2 is traveling.

[0029] Devices that acquire position and attitude information include, for example, Global Navigation Satellite System (GNSS) receivers, Inertial Measurement Units (IMUs), and Distance Measuring Instruments (DMIs). GNSS measures position by simultaneously receiving radio waves from multiple satellites and measuring the distance from each satellite. IMUs are equipped with a three-axis angular velocity (gyro) sensor and a three-axis acceleration sensor, and detect three-dimensional inertial motion (translational and rotational motion in three orthogonal axes). DMIs calculate distance traveled by measuring the number of wheel rotations.

[0030] The road condition acquisition device 3 calculates the self-position of the survey vehicle 2 by combining, for example, the positioning results of the GNSS receiver, the inertial positioning results of the IMU, and distance information from the DMI. Here, it is preferable to use network-based RTK (Real-Time Kinematics) technology for GNSS-based positioning. RTK is a measurement method known as "relative positioning." RTK can obtain high-precision positioning information (with an error of, for example, a few centimeters) by correcting for deviations by combining not only satellite data but also correction information transmitted from a reference station installed on the ground.

[0031] Devices that acquire information about the shape and conditions of roads and their surroundings include, for example, digital cameras and laser scanners. Digital cameras acquire the conditions around the road as image data. It is preferable to use a panoramic camera that can acquire 360-degree images. Digital cameras take continuous images at intervals that allow for continuous capture of the road and roadside. Digital cameras may also be capable of capturing video. Video is made up of multiple images (frames). Laser scanners acquire spatial position information of objects by irradiating a laser. It is preferable to use a 3D laser scanner that can acquire the 3D coordinates (point cloud data) of the surface shape of an object by irradiating a laser in a radial pattern.

[0032] The road condition acquisition device 3 converts the coordinates of the point cloud data into public coordinates using location information obtained by GNSS or the like. In addition, in order to visualize the surrounding shape in an easy-to-understand manner, the road condition acquisition device 3 can generate colored point cloud data by adding color information from color images taken with a digital camera to the point cloud data. This makes the point cloud data appear as if it were a three-dimensional image.

[0033] The drivability prediction device 4 shown in FIG. 1 performs a vehicle driving simulation in a virtual space realized by a computer, and predicts the drivability of the vehicle. The drivability prediction device 4 performs a driving simulation by reproducing roads and roadsides in a virtual space using point cloud data acquired by the road condition acquisition device 3. In this embodiment, it is assumed that the drivability of a special vehicle such as a trailer is to be predicted, and a driving simulation is performed for the special vehicle. The special vehicle carries long and large parts (one example is the blades of a wind turbine for power generation) in its loading platform, and the drivability including the loaded items is predicted.

[0034] Fig. 2 is a diagram illustrating an example of a hardware configuration of a travelability prediction device 4 according to an embodiment. The travelability prediction device 4 illustrated in Fig. 2 is, for example, an information processing device (computer) such as a server or a personal computer (PC). The travelability prediction device 4 includes, for example, a control unit 11, a memory unit 17, a communication IF (interface) 14, and an input / output IF 15. The memory unit 17 includes, for example, a main memory unit 12 and an auxiliary memory unit 13. These components of the travelability prediction device 4 are connected to each other via a bus 16.

[0035] The control unit 11 is configured with, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc., and processes instructions from software, controls memory, etc. The control unit 11, for example, deploys a program stored in the auxiliary storage unit 13 in an executable manner in the working area of ​​the main storage unit 12, and executes the program, thereby realizing various functions related to the prediction of drivability.

[0036] The main memory unit 12 is configured with, for example, a flash memory, a RAM (Random Access Memory), a ROM (Read Only Memory), etc. The main memory unit 12 temporarily stores programs executed by the control unit 11, data processed by the control unit 11, etc.

[0037] The auxiliary storage unit 13 is configured with, for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), or the like. The auxiliary storage unit 13 stores programs and data necessary for predicting drivability. Note that some or all of the information stored in the auxiliary storage unit 13 may be stored in a location other than the drivability prediction device 4 and acquired as needed via the communication IF 14. The auxiliary storage unit 13 may also be a CD, DVD, BD, USB (Universal Serial Bus) memory, SD (Secure Digital) memory card, or the like.

[0038] The communication IF 14 is an interface with a cable or a network to which the drivability prediction device 4 is connected. The communication IF 14 enables communication with other devices via various networks, such as a public network such as the Internet, a wireless network such as a mobile phone network, a dedicated network such as a VPN (Virtual Private Network), and a LAN (Local Area Network).

[0039] The input / output IF15 is an interface for inputting and outputting data to and from devices connected to the travel feasibility prediction device 4. Input devices such as a keyboard, a pointing device such as a touch panel or a mouse, and a microphone are connected to the input / output IF15. The travel feasibility prediction device 4 receives operation instructions and the like from an operator who operates the input device via the input / output IF15. In addition, output devices such as a display device such as a liquid crystal panel (LCD: Liquid Crystal Display) or an organic EL panel (EL: Electroluminescence), a printer, a speaker, and the like are connected to the input / output IF15.

[0040] Functions realized by the travelability prediction device 4 according to the embodiment will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of the functional configuration of the travelability prediction device 4 according to the embodiment. 3, the travelability prediction device 4 includes an acquisition unit 21, a model creation unit 22, a travel verification unit 23, a travelability prediction unit 24, and a prediction result output unit 25. These functions are realized, for example, by executing a program.

[0041] The acquisition unit 21 shown in FIG. 3 acquires point cloud data of the road and roadside for which drivability is predicted. The point cloud data includes, for example, data relating to the road surface, the ground surface of the roadside, and objects present along the roadside (e.g., earth and sand, trees, utility poles, road signs, etc.). The ground surface of the roadside includes an area where a vehicle will travel if it crosses the road, and may be subject to widening work. The objects present along the road include obstacles that may hinder vehicle passage, and may be subject to removal work. Public coordinates are assigned to the point cloud data. The acquired point cloud data is stored in the storage unit 17.

[0042] Furthermore, the acquisition unit 21 acquires the vehicle model, the cargo model, and the traveling trajectory data, and stores the acquired vehicle model, the cargo model, and the traveling trajectory data in the storage unit 17. There are no particular limitations on the targets from which the acquisition unit 21 acquires the vehicle model, the cargo model, and the traveling trajectory data. Instead of the acquisition unit 21 acquiring the vehicle model, the cargo model, and the traveling trajectory data from another device, the drivability prediction device 4 may create the vehicle model, the cargo model, and the traveling trajectory data itself.

[0043] The vehicle model is three-dimensional shape data that reproduces a vehicle for which drivability is predicted. The cargo model is three-dimensional shape data that reproduces cargo carried on a vehicle for which drivability is predicted. The vehicle model and cargo model are data that are shape modeled in a virtual three-dimensional space. Specifically, they are three-dimensional shape models that represent the range of surface shapes visible to humans using a plurality of polygons (e.g., polygonal polygons). Note that the format of these data is not particularly limited as long as they can be expressed as a three-dimensional shape model using polygons.

[0044] FIG. 4 shows an example of a vehicle model and a load model. FIG. 4 is an image diagram of the vehicle model and the load model. The modeled vehicle 8 is a special vehicle such as a trailer. The vehicle 8 has a loading platform 8a. The modeled load 9 is a blade of a wind turbine for power generation. The load 9 is placed on the loading platform 8a.

[0045] The travel trajectory data is data relating to the travel trajectory of the vehicle 8 when the vehicle 8 travels on a road for which drivability is predicted. The travel trajectory data may be either three-dimensional data or two-dimensional data (data excluding height components). The travel trajectory data includes a trajectory that crosses a border from a road when the curvature or width of the road is insufficient for the vehicle 8 to travel on.

[0046] The model creation unit 22 shown in FIG. 3 uses the point cloud data acquired by the acquisition unit 21 to create a road environment model that reproduces the road and roadside along which the vehicle travels. The road environment model includes, for example, a road model that reproduces the road surface, a ground model that reproduces the ground surface of the roadside, and a roadside object model that reproduces objects along the road. The model creation unit 22 creates the road environment model, for example, by mesh-converting the point cloud data. Specifically, the model creation unit 22 extracts only the necessary portion of the point cloud data and divides the extracted portion into polygons using various CAD software to create a 3D shape model. An example of the road environment model is shown in FIG. 5. FIG. 5 is an image diagram of the road environment model.

[0047] 3 performs a vehicle running simulation in a virtual space. An example of the running simulation performed by the running verification unit 23 is as follows. The driving verification unit 23 acquires two-dimensional plane point cloud data extracted by excluding height components from point cloud data (which may be a road environment model). The driving verification unit 23 may also acquire separately prepared two-dimensional plane point cloud data relating to road surfaces and roadside objects. Next, the driving verification unit 23 overlays the vehicle driving trajectory on the two-dimensional plane point cloud data, thereby linking the road environment model with the driving trajectory. Next, the driving verification unit 23 creates a three-dimensional VR space based on superimposed data in which the driving trajectory of the vehicle is superimposed on the road environment model, and drives the vehicle carrying the cargo along the driving trajectory in the three-dimensional VR space.

[0048] FIG. 6 shows an example of a driving simulation. FIG. 6 is an image diagram of the driving simulation. As shown in FIG. 6, a road 7a, a roadside ground surface 7b, and roadside objects 7c are reproduced in a virtual space, and a driving trajectory 7d is set on the road 7a (parts of which may extend beyond the roadside). In the virtual space, the uneven shapes of the road 7a and the roadside ground surface 7b are also reproduced, and for example, there is a convex portion 7e that is higher than other areas. A vehicle 8 carrying a load 9 travels along the driving trajectory 7d. In this embodiment, the center in the lateral direction of the axle of the front wheel 8b (see FIG. 4) of the vehicle 8 travels along the driving trajectory 7d.

[0049] The drivability prediction unit 24 shown in FIG. 3 predicts the drivability of the vehicle 8 based on a driving simulation. The drivability prediction unit 24 determines at least one of a range 31 (see FIG. 6, hereinafter referred to as "border crossing range 31") in which the vehicle 8 crosses the road 7a and an obstacle 32 (see FIG. 6) that interferes with the vehicle 8 (including the loading platform 8a and the cargo 9). The obstacle 32 includes an object 7c that exists along the road and a convex portion 7e on the road surface or the ground along the road. In FIG. 6, the border crossing range 31 exists at a corner. Also, in FIG. 6, an obstacle 32 that may come into contact with the vehicle 8 or the cargo 9 exists within the border crossing range 31.

[0050] The prediction result output unit 25 outputs a prediction result of predicting the drivability of the vehicle 8. The prediction result of the drivability includes at least one of border crossing range information regarding the border crossing range 31 and interference range information regarding the interfering obstacle 32. The prediction result output unit 25 may output information regarding a driving simulation in addition to the drivability.

[0051] The drivability prediction unit 24 may calculate the area of ​​the border crossing range 31 and the quantity (or volume) of the obstacles 32, and include the calculated information in the drivability prediction result. The drivability prediction unit 24 may also set a widening cost per unit area for widening work in advance, and calculate the cost required for the widening work by multiplying the calculated border crossing range 31 by the widening cost per unit area. Similarly, the drivability prediction unit 24 may set a removal cost per unit quantity (or unit volume) for removal work in advance, and calculate the cost required for the removal work by multiplying the calculated obstacles 32 by the removal cost per unit quantity (or unit volume). In this case, the prediction result output unit 25 may include information on the calculated costs required for the widening work and the removal work in the drivability prediction result.

[0052] The display device 5 shown in Fig. 1 is, for example, a tablet terminal or a smartphone. The display device 5 is capable of fusing a 3D shape model or a simulation result created on a computer with an image of the real world using AR technology and displaying the fusing result. In this embodiment, the display device 5 displays a fused image in which a photographed image of a road and roadside is fusing with a predicted result of drivability.

[0053] As shown in FIG. 1, the display device 5 includes, for example, a display unit 5a, an image capturing unit 5b, and an image capturing condition acquiring unit 5c. The display unit 5a is a component capable of displaying image data, and may be, for example, a display, such as a liquid crystal panel or an organic EL panel.

[0054] The photographing unit 5b is, for example, a digital camera, and is capable of photographing either or both images and videos. The photographing unit 5b acquires the road and the surrounding conditions of the road as image data. In this embodiment, a case will be described in which the photographing unit 5b photographs videos. The videos photographed by the photographing unit 5b can be displayed in real time on the display unit 5a.

[0055] The photographing situation acquisition unit 5c acquires information related to the photographing situation. The photographing situation acquisition unit 5c acquires, for example, position information, attitude information, angle of view information (or focal length information) of the display device 5. The photographing situation acquisition unit 5c includes, for example, a receiver for a global navigation satellite system (GNSS), an inertial measurement unit (IMU), and the like. For positioning using the GNSS, it is preferable to use network-based RTK technology.

[0056] Functions realized by the display device 5 according to the embodiment will be described with reference to Fig. 7. Fig. 7 is a diagram showing an example of the functional configuration of the display device 5 according to the embodiment. As shown in Fig. 7, the display device 5 includes a storage unit 5d, a control unit 5e, and a communication IF 5f in addition to the components described so far.

[0057] The storage unit 5d is configured with a flash memory, a RAM (Random Access Memory), a ROM (Read Only Memory), a HDD (Hard Disk Drive), an SSD (Solid State Drive), etc. Note that some or all of the information stored in the storage unit 5d may be stored in a location other than the display device 5 and may be acquired as needed via the communication IF 5f. The storage unit 5d may also be a CD, a DVD, a BD, a USB memory, an SD memory card, etc.

[0058] The control unit 5e is configured with, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc., and processes instructions from software, controls memory, etc. The control unit 5e, for example, deploys programs stored in the storage unit 5d in an executable manner in a work area, and executes the programs to realize various functions related to displaying the prediction results of drivability.

[0059] The communication IF 5f is an interface with a cable or network to which the display device 5 is connected. The communication IF 5f enables communication with other devices via various networks, such as a public network such as the Internet, a wireless network such as a mobile phone network, a dedicated network such as a VPN (Virtual Private Network), and a LAN (Local Area Network).

[0060] 7, the display device 5 includes an acquisition unit 41 and a driving possibility display control unit 42. The driving possibility display control unit 42 includes a fusion image creation unit 43 and a display control unit 44. These functions are realized, for example, by executing a program.

[0061] The acquisition unit 41 shown in FIG. 7 acquires the results of the driving simulation from the drivability prediction device 4. The results of the driving simulation include, for example, a vehicle model, trajectory information showing the trajectory of specific parts of the vehicle (for example, the positions of the wheels or the outermost edge), and a prediction result of drivability. The prediction result of drivability includes at least one of border crossing range information about the border crossing range 31 and interference range information about the interfering obstacle 32. If the vehicle is carrying a load, the results of the driving simulation may include, for example, a load model that reproduces the load and trajectory information showing the trajectory of specific parts of the load (for example, the rear end of the load). The acquired results of the driving simulation are stored in the memory unit 5d.

[0062] The acquisition unit 41 also acquires the video captured by the image capture unit 5b and information about the shooting conditions acquired by the shooting condition acquisition unit 5c. The captured video and the information about the shooting conditions are linked by some information (for example, time). The captured video and the information about the shooting conditions are stored in the storage unit 5d.

[0063] 7 uses AR technology to fuse a 3D shape model or simulation result created on a computer with an image or video of the real world and display it. The drivability display control unit 42 has, for example, a fused image creation unit 43 and a display control unit 44. Note that a device separate from the display device 5 (referred to as a "drivability display control device") may have functions related to the drivability display control unit 42, and the drivability display control device may control the display of images on the display device 5.

[0064] The fusion image creation unit 43 creates a fusion image by fusing the results of the driving simulation with images of the road and roadside. The results of the driving simulation include, for example, a vehicle model, trajectory information showing the trajectory of specific parts of the vehicle (for example, the positions of the wheels and the outermost edge), a cargo model that reproduces the cargo, trajectory information showing the trajectory of specific parts of the cargo (for example, the rear end of the cargo), and a predicted result of drivability.

[0065] The display control unit 44 displays the fusion image on the display unit 5a. The display control unit 44 has a function to make the predicted results of the driving possibility easy to recognize. For example, it makes it possible to select ON / OFF display of the results of the driving simulation.

[0066] The display of the fusion image by the display control unit 44 will be described with reference to Fig. 8 and Fig. 9. Fig. 8 is a display screen of the fusion image when the results of the driving simulation are set to "OFF (non-display)." Fig. 9 is a display screen of the fusion image when the results of the driving simulation are set to "ON (display)." In Fig. 8, the results of the driving simulation are not displayed, whereas in Fig. 9, the results of the driving simulation are displayed.

[0067] As shown in FIGS. 8 and 9, the fusion image display screen 60 has a fusion image display area 61, a menu icon 62, a cross-section display adjustment section 63, and a transparency adjustment section 64.

[0068] The fusion image display area 61 is an area for displaying a fusion image. Various settings can be made by selecting the menu icon 62. For example, the results of a driving simulation can be read by operating the menu icon 62. The cross-section display adjustment unit 63 is an operation unit for realizing a cross-section display of the results of the driving simulation (including various models). By sliding the cross-section display adjustment unit 63 left and right, it becomes possible to view a cross-section of the results of the driving simulation from the position of the display device 5. The cross-section is always displayed perpendicular to the viewing direction.

[0069] The transparency adjustment unit 64 is an operation unit for realizing transparent, semi-transparent, or opaque display of the driving simulation results. The transparency of the driving simulation results can be changed by sliding the transparency adjustment unit 64 left or right. For example, as shown in FIG. 8, by moving the transparency adjustment unit 64 to the left end, the driving simulation results are displayed transparently, and the display of the driving simulation results disappears (becomes unrecognizable) from the display screen 60. On the other hand, as shown in FIG. 9, by moving the transparency adjustment unit 64 to the right, the driving simulation results gradually become opaque, and become visible. Then, by moving the transparency adjustment unit 64 to the right end, the driving simulation results are displayed opaque.

[0070] 8, the results of the driving simulation are displayed transparently on display screen 60A, so only the image captured by image capturing unit 5b is displayed. Display screen 60A displays the actual road surface, the ground along the road, and objects and protrusions along the road.

[0071] 9, the results of the driving simulation are displayed opaquely, so both the image captured by the image capturing unit 5b and the results of the driving simulation are displayed. In addition to the actual road surface, the ground along the road, objects and protrusions along the road, the following items (A) to (C) are displayed on the display screen 60B. (A) A model of a vehicle 8, a trajectory 71 of the front wheel 8b, a trajectory 72 of the rearmost wheel 8c of the loading platform 8a, and a trajectory 73 of the outermost edge 8d of the vehicle 8 (see Figure 4). (a) Modeled payload 9, trajectory 81 of the rearmost end 9a of the payload (c) The predicted results of the drivability include the range 31 in which the vehicle 8 crosses the road 7a, and obstacles 32 that interfere with the vehicle 8 (including the loading platform 8a and the load 9).

[0072] <Drivability display method by drivability display system according to embodiment> 10 (and also FIGS. 1 to 9 as appropriate), a description will be given of a drivability display method by the drivability display system 1. FIG. 10 is a schematic diagram showing the flow of the drivability display method by the drivability display system 1. As shown in FIG. 10, the travel feasibility display method mainly includes a road condition acquisition step S10, an acquired information storage step S20, a travel feasibility prediction step S30, and a fusion image display step S40.

[0073] In the road condition acquisition step S10, the road condition acquisition device 3 acquires point cloud data of roads and roadsides. Public coordinates are assigned to the point cloud data. In the acquired information storage step S20, the acquired point cloud data information is stored in a storage device with which the drivability prediction device 4 can communicate. Note that the acquired information storage step S20 can be omitted, and data may be transmitted from the road condition acquisition device 3 to the drivability prediction device 4.

[0074] In the drivability prediction step S30, the drivability prediction device 4 recreates roads and roadsides in a virtual space using point cloud data and performs a vehicle driving simulation in the virtual space. The drivability prediction device 4 also predicts the drivability of the vehicle based on the results of the driving simulation. In the fusion image display step S40, the display device 5 displays a fusion image in which the predicted result of drivability is fused with an image of the road and the roadside taken.

[0075] As described above, the drivability display system 1 according to this embodiment displays a fusion image in which the results of the driving simulation in virtual space (including the predicted results of drivability) are associated with the real world. Therefore, it is possible to determine at a glance whether a vehicle can be driven at an actual construction site.

[0076] Furthermore, the driving possibility display system 1 displays the trajectory of the vehicle at specific locations and the trajectory of the cargo at specific locations, making it easier to visualize the image of the vehicle driving and the range affected by the cargo, making it easier to determine the vehicle's driving possibility.

[0077] Furthermore, according to the driving feasibility display system 1, the transparency adjustment unit 64 is displayed together with the fusion image, and the transparency of the driving simulation results (including the predicted results of driving feasibility) is changed and displayed in response to the operation of the transparency adjustment unit 64. This allows the results of the simulation in the virtual space to be compared with the real world, making it easier to determine the driving feasibility of the vehicle. [Explanation of symbols]

[0078] 1. Drivability Display System 2 Survey vehicles 3 Road condition acquisition device 4. Drivability prediction device 5 Display device 5a Display section 5b Photography Department 5c Shooting status acquisition unit 5d storage section 5e Control Unit 5F Communication Interface 8 vehicles 8a Cargo bed 9. Cargo 11 Control section 12 Main memory 13 Auxiliary storage 14 Communication Interface 15 Input / Output Interface 16 Bus 17 Memory section 21 Acquisition Department 22 Model Creation Department 23 Driving Verification Department 24. Drivability Prediction Unit 25 Prediction result output section 41 Acquisition Department 42 Driving possibility display control unit 43 Fusion Image Creation Department 44 Display control unit 60,60A,60B display screen 61 Fusion image display area 62 menu icons 63 Cross section display adjustment section 64 Transparency adjustment section

Claims

1. A drivability display system that displays information about the drivability of a vehicle on a road to be traveled, a road condition acquisition device for acquiring point cloud data of the road and surrounding objects along the road; a drivability prediction device that uses the point cloud data to recreate the road and the roadside in a virtual space and predicts drivability by performing a vehicle travel simulation in the virtual space; and a display device that displays a fusion image obtained by fusing an image of the road and the roadside with a prediction result of drivability, the prediction result includes at least one of border crossing range information indicating a range in which the vehicle crosses the road and interference range information indicating a range in which the vehicle interferes with the road; A driving possibility display system characterized by:

2. When acquiring the point cloud data, the road condition acquisition device acquires coordinates of the road and the roadside, The travel possibility prediction device outputs the prediction result reflecting the coordinates, the display device acquires a photographing position when photographing the road and the roadside, and creates the fusion image based on the coordinates of the prediction result and the photographing position.

2. The drivability display system according to claim 1.

3. The travel possibility prediction device a model creation unit that recreates the road and the roadside in a virtual space using point cloud data of the road and the roadside; a driving verification unit that performs a driving simulation of the vehicle in the virtual space; a travel possibility prediction unit that predicts travel possibility by determining at least one of a range in which the vehicle crosses the road and a range in which the vehicle interferes with the road, 2. The drivability display system according to claim 1.

4. the prediction result further includes a vehicle model that reproduces the vehicle and trajectory information that indicates a trajectory of a specific portion of the vehicle.

2. The drivability display system according to claim 1.

5. the vehicle has a load; the prediction result further includes a vehicle model that reproduces the vehicle, a load model that reproduces the load, and trajectory information that indicates a trajectory of a specific portion of the load.

2. The drivability display system according to claim 1.

6. the display device displays a transparency adjustment unit together with the fusion image, and changes the transparency of the prediction result in response to an operation of the transparency adjustment unit and displays the result.

2. The drivability display system according to claim 1.

7. A drivability display method for displaying information about the drivability of a vehicle on a road to be traveled, comprising: a road condition acquisition step of acquiring point cloud data of the road and roadside; a drivability prediction step of reproducing the road and the roadside in a virtual space using the point cloud data and predicting drivability by performing a vehicle running simulation in the virtual space; a display step of displaying a fusion image obtained by fusing the predicted result of drivability with an image of the road and the roadside, the prediction result includes at least one of border crossing range information indicating a range in which the vehicle crosses the road and interference range information indicating a range in which the vehicle interferes with the road; A method for displaying drivability.

8. A travel possibility display control device that displays information about the travel possibility of a vehicle on a road on which a vehicle is to travel on a display unit, a storage unit that stores a predicted result of drivability obtained by performing a driving simulation of the vehicle in a virtual space; a fusion image creation unit that creates a fusion image by fusing the prediction result with an image of the road and roadside; a display control unit that displays the fusion image on the display unit, the prediction result includes at least one of border crossing range information indicating a range in which the vehicle crosses the road and interference range information indicating a range in which the vehicle interferes with the road; A driving possibility display control device characterized by the above.

9. A travel possibility display control program that displays information about the travel possibility of a vehicle on a road on which a vehicle is to travel on a display unit, a computer having a storage unit that stores a predicted result of drivability obtained by performing a driving simulation of the vehicle in a virtual space, a fusion image creation unit that creates a fusion image by fusing the prediction result with an image of the road and roadside; a display control unit that causes the fusion image to be displayed on the display unit; the prediction result includes at least one of border crossing range information indicating a range in which the vehicle crosses the road and interference range information indicating a range in which the vehicle interferes with the road; A driving possibility display control program characterized by:

Images