Site map generation device, supply chain management system, and site map generation method

Abstract

To enable, by only photographing a site, a highly accurate site map to be generated which allows a specific state of the site to be appropriately determined, and allows a user to appropriately check the specific state of the site.SOLUTION: Photographed image data of a color camera is acquired; an NeRF model (machine learned model) for generating a free viewpoint image is constructed on the basis of the photographed image data; information regarding a risk condition at the site is acquired on the basis of the free viewpoint image generated by the NeRF model; and a disaster prevention map (site map) visualizing a specific state of the site is generated on the basis of the information regarding the risk condition of the site. Especially, by determining the specific state of the site from the free viewpoint image according to a viewpoint of a specific direction by using an image recognition model constructed by the machine learning, information regarding the specific state is acquired. Alternatively, an edition terminal is made to display the free viewpoint image, and according to a user operation inputting the specific state of the site to the free viewpoint image, information regarding the specific state is acquired.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a local map generation device and a local map generation method that generate a local map that visualizes specific local conditions based on the detection results of sensors mounted on a mobile object, and a supply chain management system that presents optimal delivery routes based on supply chain management. [Background technology]

[0002] In recent years, natural disasters such as heavy rain and earthquakes have been occurring frequently, creating a demand for technology to efficiently create disaster prevention maps (disaster prediction maps) that show locations where disasters are predicted to occur.

[0003] A known technology for creating map information related to such disasters involves a drone equipped with a camera flying over an area targeted for disaster prevention measures, photographing the area with the camera, thereby collecting images of the area, and then generating three-dimensional information related to the shape of the land in the area, specifically 3D mesh data using the TIN (Triangulated Irregular Network) method, based on the images (see Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2017-201261 Summary of the Invention [Problem to be solved by the invention]

[0005] According to conventional technology, highly accurate location information is acquired based on ground control points (GCPs) installed on the ground, and 3D information about the shape of the land is generated with high accuracy based on the location information. This allows users to quickly and accurately grasp changes in the condition of a target area, and quickly determine, for example, the occurrence of a disaster.

[0006] However, when creating a disaster prevention map to show disaster-prevention personnel where disasters are predicted to occur, not only changes in the shape of the area but also changes in color are important factors for determining the risk of disaster, and it is desirable to be able to observe the condition of the area from various angles. However, conventional technology has not taken such requests into consideration, and has had the problem of being unable to properly determine the risk of disaster. This applies whether the risk of disaster is determined visually by a person or by a computer.

[0007] Furthermore, conventional technology requires that control points be installed on the ground, which requires workers to visit the site (mountain slopes, cliffs, etc.) to install these control points. Furthermore, when local governments and other organizations create disaster prevention maps covering vast areas, it is practically difficult to install control points over such a large area. Therefore, it is desirable to be able to create highly accurate disaster prevention maps simply by taking photographs of the site, without installing control points on site.

[0008] Therefore, the main object of the present invention is to provide a local map generation device and a local map generation method that can appropriately determine the specific conditions of a local area simply by photographing the local area, and generate a highly accurate local map that allows the user to appropriately confirm the specific conditions of the local area. [Means for solving the problem]

[0009] The local map generation device of the present invention is a local map generation device equipped with a processor that performs processing to generate a local map that visualizes the specific state of the local area based on the detection results of a sensor mounted on a mobile body, and the processor is configured to acquire captured image data as the detection results of a visible camera serving as the sensor, generate a free viewpoint image based on the captured image data, acquire information regarding the specific state of the local area from the free viewpoint image, and generate a local map that visualizes the specific state of the local area based on the information regarding the specific state.

[0010] The supply chain management system of the present invention also includes the local map generation device and a supply chain management device that presents optimal delivery routes based on supply chain management, and the supply chain management device acquires the local map that visualizes a specific local condition from the local map generation device, builds a supply chain management prediction engine through machine learning using the local map as learning data, and uses the supply chain management prediction engine to generate optimal delivery routes predicted under normal circumstances and optimal delivery routes predicted in the event of a disaster.

[0011] In addition, the local map generation method of the present invention is a local map generation method that causes a processor to perform a process of generating a local map that visualizes a specific local condition based on the detection results of a sensor mounted on a moving body, and is configured to obtain captured image data as the detection results of a visible camera as the sensor, generate a free viewpoint image based on the captured image data, obtain information regarding the specific local condition from the free viewpoint image, and generate a local map that visualizes the specific local condition based on the information regarding the specific condition. [Effects of the Invention]

[0012] According to the present invention, a free viewpoint image with the same level of resolution as the captured image is generated from a viewpoint in any direction, so that the specific conditions of the site can be properly determined simply by photographing the site, and a highly accurate site map can be generated that allows a user (e.g., disaster prevention personnel) to properly check the specific conditions of the site. [Brief explanation of the drawings]

[0013] [Figure 1] Overall configuration diagram of a disaster prevention map generation system according to the first embodiment [Figure 2] Block diagram showing the general configuration of the camera [Figure 3] Block diagram showing the schematic configuration of the data processing terminal and disaster prevention map server [Figure 4] An explanatory diagram showing the outline of the risk reflection process performed by the disaster prevention map server [Figure 5] A flow diagram showing the processing procedures performed by the camera, data processing terminal, and disaster prevention map server. [Figure 6] FIG. 1 is a flowchart showing the procedure of position and orientation estimation processing performed in a data processing terminal. [Figure 7] Flow diagram showing the procedure for generating disaster prevention map data performed by the disaster prevention map server [Figure 8] An explanatory diagram showing the editing screen displayed on the editing terminal when creating a new file. [Figure 9] An explanatory diagram showing the editing screen displayed on the editing terminal when creating a new file. [Figure 10] FIG. 10 is an explanatory diagram showing a viewing screen displayed on a viewing terminal. [Figure 11] FIG. 10 is an explanatory diagram showing a viewing screen displayed on a viewing terminal. [Figure 12] An explanatory diagram showing the editing screen displayed on the editing terminal during updating [Figure 13] An explanatory diagram showing the editing screen displayed on the editing terminal during updating [Figure 14] FIG. 10 is an explanatory diagram showing a vehicle as a moving body on which an imaging device according to a modification of the first embodiment is mounted; [Figure 15] Overall configuration diagram of a delivery route presentation system according to a second embodiment [Figure 16] Block diagram showing the general configuration of the SCM server [Figure 17] FIG. 10 is an explanatory diagram showing a viewing screen displayed on a viewing terminal. DETAILED DESCRIPTION OF THE INVENTION

[0014] The first invention made to solve the above problem is a local map generation device equipped with a processor that performs processing to generate a local map that visualizes a specific local condition based on the detection results of a sensor mounted on a moving body, wherein the processor acquires captured image data as the detection results of a visible camera serving as the sensor, generates a free viewpoint image based on the captured image data, acquires information about the specific local condition from the free viewpoint image, and generates a local map that visualizes the specific local condition based on the information about the specific condition.

[0015] This allows free viewpoint images with the same level of resolution as the captured image to be generated from any viewpoint, so that the specific conditions of the site can be properly determined simply by photographing the site, and a highly accurate site map can be generated that allows users (e.g., disaster prevention personnel) to properly check the specific conditions of the site.

[0016] In a second aspect of the present invention, the processor acquires point cloud data as a detection result of a three-dimensional sensor serving as the sensor, and performs scaling of the captured image data based on the point cloud data.

[0017] This allows for highly accurate scaling of captured image data, resulting in the generation of highly accurate local maps. In this case, a point cloud image generated from the point cloud data may be presented to the user. Furthermore, local measurements may be performed based on the point cloud data. Furthermore, the 3D sensor may be a LiDAR sensor.

[0018] In addition, a third invention is configured such that the processor uses an image recognition model constructed by machine learning to determine a specific state of the site from the free viewpoint image taken from a viewpoint in a predetermined direction, and acquires information about the specific state.

[0019] This allows appropriate acquisition of information relating to the specific state of the local area. In this case, a risk state (degree of risk) in which a disaster may occur may be determined as the specific state.

[0020] In addition, a fourth invention is configured such that the processor displays an editing screen including the free viewpoint image on a display device, and acquires information about the specific state in response to a user's operation of inputting a specific state of the local area for the free viewpoint image on the editing screen.

[0021] This allows the user to visually view the high-resolution free viewpoint image and appropriately determine the specific state of the site, thereby enabling appropriate acquisition of information regarding the specific state. In this case, the user may determine the danger state (degree of danger) of a disaster occurring as the specific state.

[0022] In addition, a fifth invention is configured such that the processor generates the free viewpoint image from the specified viewpoint in response to a user operation to specify an arbitrary viewpoint, and causes a display device to display a viewing screen including the free viewpoint image from the specified viewpoint.

[0023] This allows the user to easily view a high-definition free viewpoint image from a viewpoint in any direction.

[0024] In addition, a sixth invention is configured such that the processor generates the free viewpoint image based on pre-stored initial values ​​related to the depression angle and height of the viewpoint, and displays the free viewpoint image on the display device, and in response to a user operation specifying the depression angle and height of the viewpoint, generates the free viewpoint image from a viewpoint with the specified depression angle and height, and displays the free viewpoint image on the display device.

[0025] This allows the user to easily view a free viewpoint image from a viewpoint with a standard depression angle and height, and further allows the user to easily view a free viewpoint image from a viewpoint with a desired depression angle and height.

[0026] A seventh invention includes the local map generation device and a supply chain management device that presents optimal delivery routes based on supply chain management, wherein the supply chain management device acquires the local map that visualizes a specific local condition from the local map generation device, builds a supply chain management prediction engine through machine learning using the local map as learning data, and uses the supply chain management prediction engine to generate optimal delivery routes predicted under normal circumstances and optimal delivery routes predicted in the event of a disaster.

[0027] With this system, when a disaster occurs, the optimal delivery route predicted for the disaster is generated in a short time using a supply chain management prediction engine and quickly presented to the delivery manager.

[0028] Furthermore, an eighth invention is a local map generation method that causes a processor to perform processing to generate a local map that visualizes a specific state of the local area based on the detection results of a sensor mounted on a moving body, and is configured to obtain captured image data as the detection results of a visible camera as the sensor, generate a free viewpoint image based on the captured image data, obtain information about the specific state of the local area from the free viewpoint image, and generate a local map that visualizes the specific state of the local area based on the information about the specific state.

[0029] As with the first invention, this allows a free viewpoint image with the same level of resolution as the captured image to be generated from a viewpoint in any direction, so that the specific conditions of the site can be properly determined simply by photographing the site, and a highly accurate site map can be generated that allows a user (e.g., disaster prevention personnel) to properly check the specific conditions of the site.

[0030] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0031] (First embodiment) FIG. 1 is a diagram showing the overall configuration of a disaster prevention map generation system according to the first embodiment.

[0032] The disaster prevention map generation system generates a disaster prevention map that visualizes local danger conditions in an area subject to disaster prevention measures. The disaster prevention map generation system includes a camera 1, a data processing terminal 2, a disaster prevention map server 3 (local map generation device), an editing terminal 4 (display device), and a viewing terminal 5 (display device).

[0033] The photographing device 1 includes a sensor unit 11. The sensor unit 11 includes a color camera 12 (visible light camera) and a LiDAR sensor 13 (three-dimensional sensor). The photographing device 1 is mounted on a drone 6 (a moving object). When the drone 6 flies above a target area, the photographing device 1 performs 3D sensing (three-dimensional measurement) using the color camera 12 and the LiDAR sensor 13.

[0034] The color camera 12 detects visible light to capture an image of an object, and outputs color captured image data in, for example, the RGB format.

[0035] The LiDAR sensor 13 uses LiDAR (Light Detection and Ranging) technology to detect the distance to an object by reflecting laser light, and outputs point cloud data as the detection result.

[0036] The data processing terminal 2 receives the detection data from the photographing device 1 detected by the sensor unit 11 of the photographing device 1, processes the detection data, and transmits the processed data to the disaster prevention map server 3.

[0037] The disaster prevention map server 3 is configured as a cloud computer. The disaster prevention map server 3 generates 3D data about the target area using data collected by 3D sensing of the target area using the camera 1. Specifically, the disaster prevention map server 3 constructs a Neural Radiance Fields (NeRF) model that generates images (free viewpoint images) from any viewpoint different from the actually captured images, and generates NeRF mesh data. The disaster prevention map server 3 also integrates point cloud data from the LiDAR sensor 13 to generate point cloud data for the entire target area.

[0038] The editing terminal 4 is configured as a PC. The editing terminal 4 is operated by a creator who belongs to the organization that creates the disaster prevention map. The creator can operate the editing terminal 4 to access the disaster prevention map server 3 and perform editing operations on the disaster prevention map generated by the disaster prevention map server 3.

[0039] Here, the person in charge of creating the map can visually check the free viewpoint image displayed on the editing terminal 4 to determine the degree of danger at each point within the target area, thereby inputting the degree of danger at each point (manual input mode). Also, the disaster prevention map server 3 can obtain the degree of danger at each point within the target area by performing image recognition processing on the free viewpoint image (data analysis mode). The disaster prevention map server 3 generates a disaster prevention map based on the degree of danger at each point within the target area.

[0040] The viewing terminal 5 is configured as a PC. The viewing terminal 5 is operated by disaster prevention personnel, i.e., personnel from the disaster prevention department of a local government, a fire station, or a police station. By operating the viewing terminal 5 to access the disaster prevention map server 3, the disaster prevention personnel can view a screen displaying a disaster prevention map generated by the disaster prevention map server 3. Specifically, a 2D disaster prevention map that visualizes the level of danger and a free viewpoint image that visualizes the level of danger are displayed on the viewing terminal 5.

[0041] In this embodiment, a disaster prevention map is generated that visualizes danger states related to natural disasters such as heavy rain and earthquakes as specific local conditions, but the present invention is not limited to such a disaster prevention map. In other words, a local condition map that visualizes specific conditions other than danger states related to natural disasters may also be generated.

[0042] In this embodiment, an image from an arbitrary viewpoint (free-viewpoint image) is generated using NeRF technology, but the technology for generating free-viewpoint images is not limited to NeRF. For example, a free-viewpoint image may be generated using 3D Gaussian Splatting technology, which is a new 3D data format that can function as both a point cloud and a photograph and has been attracting attention in recent years.

[0043] Next, we will explain the general configuration of the photographing device 1. Figure 2 is a block diagram showing the general configuration of the photographing device 1.

[0044] In addition to the sensor unit 11, the photographing device 1 also includes a communication unit 16, a storage unit 17, and a processor .

[0045] The sensor unit 11 includes a color camera 12 and a LiDAR sensor 13, as well as a satellite positioning sensor 14 and an IMU sensor 15.

[0046] The satellite positioning sensor 14 detects the position of the device itself using a satellite positioning system such as GPS, and outputs position information in a world coordinate system.

[0047] The IMU sensor 15 detects the motion state of the device itself, specifically, three-dimensional angular velocity and acceleration. Based on the detection results of the IMU sensor 15, the reference directions (north direction, vertical direction) are recognized. This improves the accuracy of position and orientation estimation for the color camera 12 and the LiDAR sensor 13.

[0048] The communication unit 16 communicates with the data processing terminal 2 using an appropriate communication method. Specifically, the communication unit 16 transmits, to the data processing terminal 2, image data captured by the color camera 12, point cloud data from the LiDAR sensor 13, position information from the satellite positioning sensor 14, and direction information from the IMU sensor 15. The communication method employed by the communication unit 16 may be, for example, wireless communication such as a mobile communication network or a wireless LAN such as Wi-Fi (registered trademark), or wired communication such as a wired LAN or USB.

[0049] The storage unit 17 stores programs and the like to be executed by the processor 18. The storage unit 17 also stores image data captured by the color camera 12, point cloud data from the LiDAR sensor 13, position information from the satellite positioning sensor 14, and direction information from the IMU sensor 15.

[0050] The processor 18 performs various processes by executing programs stored in the storage unit 17. For example, the processor 18 stores the captured image data by the color camera 12, the point cloud data by the LiDAR sensor 13, the position information by the satellite positioning sensor 14, and the direction information by the IMU sensor 15 in the storage unit 17 and controls the communication unit 16 to transmit the data to the data processing terminal 2.

[0051] 2, the sensor unit 11 includes the color camera 12, the LiDAR sensor 13, the satellite positioning sensor 14, and the IMU sensor 15. However, the sensor unit 11 may also be configured without the LiDAR sensor 13. In this case, the scale information is generated based only on the detection result of the satellite positioning sensor 14, which reduces the accuracy of the scale information, but simplifies the configuration of the sensor unit 11. The sensor unit 11 may also be configured without the IMU sensor 15.

[0052] Next, a description will be given of the general configuration of the data processing terminal 2 and the disaster prevention map server 3. Fig. 3 is a block diagram showing the general configuration of the data processing terminal 2 and the disaster prevention map server 3.

[0053] The data processing terminal 2 includes a communication unit 21, a storage unit 22, and a processor 23.

[0054] The communication unit 21 communicates with the photographic device 1 using an appropriate communication method such as a mobile communication network. Specifically, the communication unit 21 receives photographic image data, point cloud data, position information, and direction information transmitted from the photographic device 1.

[0055] The communication unit 21 also communicates with the disaster prevention map server 3 via the network. Specifically, the communication unit 21 transmits image data, position and orientation information, point cloud data, and position information related to the key frame generated by the processor 23 to the disaster prevention map server 3.

[0056] The storage unit 22 stores programs executed by the processor 23 and the like.

[0057] The processor 23 performs various processes by executing programs stored in the storage unit 22. In this embodiment, the processor 23 performs a position and orientation estimation process, a key frame extraction process, and the like.

[0058] In the position and orientation estimation process, the processor 23 estimates the position and orientation of the LiDAR sensor 13 at the time of detection based on the point cloud data at each time from the LiDAR sensor 13 received from the image capturing device 1, and obtains position and orientation information for the LiDAR sensor 13 at each time.

[0059] In addition, in the position and orientation estimation process, the processor 23 estimates the position and orientation of the color camera 12 based on the position and orientation information for the LiDAR sensor 13 at each time and the calibration results for the position and orientation between the LiDAR sensor 13 and the color camera 12, and obtains the position and orientation information for the color camera 12 at each time.

[0060] Furthermore, in the position and orientation estimation process, the processor 23 generates scale information based on the point cloud data from the LiDAR sensor 13, and performs scale adjustment on the images (frames) captured by the color camera 12 at each time based on the scale information.

[0061] In the position and orientation estimation process, the processor 23 sets the north direction and the vertical direction for the captured image (frame) at each time included in the captured image data based on the direction information from the IMU sensor 15.

[0062] In addition, if the photographing device 1 is not equipped with a LiDAR sensor 13, in the position and orientation estimation process, the processor 23 estimates the position and orientation of the color camera 12 at the time of photographing based on the images (frames) photographed by the color camera 12 at each time received from the photographing device 1, and obtains position and orientation information for the color camera 12 at each time.

[0063] In addition, if the photographing device 1 is not equipped with a LiDAR sensor 13, in the position and orientation estimation process, the processor 23 performs scale adjustment for the photographed images (frames) at each time included in the photographing data based on the position information from the satellite positioning sensor 14.

[0064] In the key frame extraction process, the processor 23 extracts key frames from the image data captured by the color camera 12 received from the image capture device 1. At this time, key frames may be extracted at regular time intervals. Alternatively, frames with many feature points may be extracted as key frames. Alternatively, frames in which the position or attitude of the image capture device 1 has changed significantly may be extracted as key frames.

[0065] The disaster prevention map server 3 includes a communication unit 31, a storage unit 32, and a processor 33.

[0066] The communication unit 31 communicates with the data processing terminal 2, the editing terminal 4, and the viewing terminal 5 via the network.

[0067] The storage unit 32 stores programs executed by the processor 33 and the like.

[0068] The processor 33 performs various processes by executing programs stored in the storage unit 32. In this embodiment, the processor 33 performs a learning process, a point cloud integration process, a free viewpoint image generation process, a risk level acquisition process, a risk level reflection process, a risk level visualization process, and the like.

[0069] In the learning process, the processor 33 constructs a NeRF model (free viewpoint image generation model, machine learning model) by deep learning using the captured image data for the key frames received from the data processing terminal 2 and the position and orientation information of those key frames as learning data.

[0070] In the point cloud integration process, the processor 33 integrates the point cloud data at each time from the LiDAR sensor 13 received from the data processing terminal 2 based on the position and orientation information received from the data processing terminal 2, and generates point cloud data as three-dimensional information of the entire area targeted for disaster prevention measures.

[0071] In the free viewpoint image generation process, the processor 33 uses a NeRF model (free viewpoint image generation model) to generate a free viewpoint image from a viewpoint specified by the user or a preset specification. At this time, viewpoint information is input to the free viewpoint image generation model, and a free viewpoint image from the required viewpoint is output from the free viewpoint image generation model.

[0072] In the risk level acquisition process, processor 33 generates a free viewpoint image of the specified viewpoint by free viewpoint image generation process in response to a user operation to specify a viewpoint, displays it on editing terminal 4, and acquires the risk level of each point (each pixel) on the free viewpoint image in response to a user operation to input the risk level of each point on the free viewpoint image.

[0073] In the risk level acquisition process, the processor 33 generates a free-viewpoint image of a predetermined viewpoint by a free-viewpoint image generation process, and performs image recognition on the free-viewpoint image to determine the risk level of each point (each pixel) on the free-viewpoint image. In the image recognition process, an image recognition model constructed by machine learning such as deep learning may be used. In the image recognition process, the risk may be quantified based on the color and shape of each point.

[0074] In the risk level acquisition process, the processor 33 generates a point cloud image from a predetermined viewpoint, performs point cloud recognition processing on the point cloud image, and determines the risk level of each point (each pixel) on the point cloud image. In the point cloud recognition processing, a point cloud recognition model constructed by machine learning such as deep learning may be used. In the point cloud recognition processing, the risk may be quantified based on the shape of each point.

[0075] In the risk reflection process, processor 33 reflects the information acquired in the risk acquisition process, i.e., the risk level of each point on the free viewpoint image and point cloud image, which are 2D data, in the risk level of each point on the NeRF mesh data and point cloud data, which are 3D data. Specifically, a risk level is set for each mesh included in the NeRF mesh data, and a risk level is set for each point included in the point cloud data.

[0076] In the risk visualization process, processor 33 reflects the risk levels set for the 3D data (NeRF mesh data, point cloud data) of the entire target area on an area map representing the local topography to generate a 2D disaster prevention map in which the risk levels are visualized. At this time, the risk levels set for the 3D data elements (each mesh in the NeRF mesh data, each point in the point cloud data) corresponding to each point in the target area are assigned to each point (each pixel) on the 2D disaster prevention map based on position information in the world coordinate system.

[0077] In the risk visualization process, processor 33 generates a free viewpoint image from a specified viewpoint with the risk visualized (expressed by differences in color, shading, etc.) based on the risk of each point set for NeRF mesh data, which is 3D data. Also, processor 33 generates a point cloud image from a specified viewpoint with the risk visualized, based on the risk of each point set for point cloud data, which is 3D data.

[0078] Next, a description will be given of the risk level reflection process performed by the disaster prevention map server 3. Fig. 4 is an explanatory diagram showing an overview of the risk level reflection process.

[0079] The processor 33 of the disaster prevention map server 3 performs processing to reflect the risk level of each point on the free viewpoint image and point cloud image, which are 2D data set by manual input by the user or by image recognition, in the risk level of each point on the NeRF mesh data and point cloud data, which are 3D data.

[0080] Specifically, as shown in Fig. 4(A), the risk level of each mesh included in the NeRF mesh data (3D data) is set based on the risk level of each point (each pixel) on the free viewpoint image (2D data). Also, as shown in Fig. 4(B), the risk level of each point included in the point cloud data (3D data) is set based on the risk level of each point (each pixel) on the point cloud image (2D data).

[0081] At this time, based on the position information in the world coordinate system from the satellite positioning sensor 14 and the direction information (north direction, vertical direction) from the IMU sensor 15, the risk level of each point on the 2D image (free viewpoint image, point cloud image) is assigned to each point in the 3D space (NeRF mesh data, point cloud data).

[0082] In this way, the risk level is set for the 3D data, NeRF mesh data and point cloud data, so even if the viewpoint of the free-viewpoint image or point cloud image is moved, the displayed state of the risk level for each point on the free-viewpoint image or point cloud image is maintained. This allows the user to understand the risk level of a point of interest and observe the condition of that point from various angles.

[0083] Next, a description will be given of the processing procedures performed by the photographing device 1, the data processing terminal 2, and the disaster prevention map server 3. Fig. 5 is a flow diagram showing the processing procedures performed by the photographing device 1, the data processing terminal 2, and the disaster prevention map server 3.

[0084] In the photographing device 1, while the mobile object moves within an area targeted for disaster prevention measures, the color camera 12 photographs various points within the target area, and the processor 18 stores the image data photographed by the color camera 12 in the storage unit 17 (ST101). At this time, the LiDAR sensor 13 detects objects at various points within the target area, and the generated point cloud data is stored in the storage unit 17. In addition, the satellite positioning sensor 14 detects the position of the own device, and the generated position information is stored in the storage unit 17. In addition, the IMU sensor 15 detects the motion state of the own device, and the generated direction information is stored in the storage unit 17.

[0085] Next, the communication unit 16 transmits the image data from the color camera 12, the point cloud data from the LiDAR sensor 13, the position information from the satellite positioning sensor 14, and the direction information from the IMU sensor 15 to the data processing terminal 2 (ST102).

[0086] In the data processing terminal 2, the communication unit 21 receives, from the photographing device 1, the image data captured by the color camera 12, the point cloud data captured by the LiDAR sensor 13, the position information captured by the satellite positioning sensor 14, and the direction information captured by the IMU sensor 15 (ST201).

[0087] Next, the processor 23 estimates the position and orientation of the color camera 12 and the LiDAR sensor 13 at each time, and obtains position and orientation information at each time regarding the image data captured by the color camera 12 and the point cloud data captured by the LiDAR sensor 13 (position and orientation estimation process) (ST202).

[0088] Next, processor 23 extracts key frames from the image data captured by color camera 12 (key frame extraction process) (ST203). Next, processor 23 aligns the extracted key frames with the world coordinate system based on the position information from satellite positioning sensor 14 (ST204).

[0089] Next, the communication unit 21 transmits the photographed image data, position and orientation information, point cloud data, and position information related to the key frame to the disaster prevention map server 3 (ST205). The position and orientation information includes direction information (north direction, vertical direction) obtained by the IMU sensor 15.

[0090] In the disaster prevention map server 3, the communication unit 31 receives the photographed image data, position and orientation information, point cloud data, and position information relating to the key frame from the data processing terminal 2 (ST301).

[0091] Next, the processor 33 generates a NeRF model (free viewpoint image generation model) by deep learning using the captured image data and position and orientation information of the key frames received from the data processing terminal 2 as learning data (learning process) (ST302).

[0092] In addition, the processor 33 integrates the point cloud data at each time from the LiDAR sensor 13 received from the data processing terminal 2 based on the position and orientation information received from the data processing terminal 2, and generates point cloud data as three-dimensional information of the entire area subject to disaster prevention measures (point cloud integration processing) (ST303).

[0093] Next, the processor 33 generates disaster prevention map data in which the risk level is visualized (disaster prevention map data generation process) (ST304). Specifically, a 2D disaster prevention map on which the risk level heat map is superimposed is generated. In addition, a risk level is set for each mesh included in the NeRF mesh data. In addition, a risk level is set for each point included in the point cloud data.

[0094] Next, processor 33 causes viewing terminal 5 to display viewing screens 71 and 81 (see FIGS. 10 and 11) (ST305). At this time, 2D disaster prevention map 69 with the risk heat map superimposed thereon is displayed on viewing screen 71. In addition, free viewpoint image 62 based on a viewpoint specified by the user is generated based on NeRF mesh data for which a risk level has been set, and displayed on viewing screen 81. In addition, a point cloud image based on a viewpoint specified by the user is generated based on point cloud data for which a risk level has been set, and displayed on viewing screen 81.

[0095] Next, a description will be given of the position and orientation estimation process (ST202 in FIG. 5) performed by the data processing terminal 2. FIG. 6 is a flowchart showing the procedure of the position and orientation estimation process.

[0096] In the data processing terminal 2, first, the processor 23 determines whether or not the image capturing device 1 is equipped with a LiDAR sensor 13 (ST401).

[0097] Here, if the image capturing device 1 is equipped with a LiDAR sensor 13 (Yes in ST401), the processor 23 estimates the position and orientation of the LiDAR sensor 13 at the time of detection based on the point cloud data of the LiDAR sensor 13 received from the image capturing device 1, and acquires position and orientation information for each time regarding the LiDAR sensor 13 (ST402). Also, based on the position and orientation information for the LiDAR sensor 13, the processor 23 estimates the position and orientation of the color camera 12 at the time of image capturing, and acquires position and orientation information for each time regarding the color camera 12 (ST403).

[0098] Next, the processor 23 performs scale adjustment for the images (frames) captured by the color camera 12 at each time based on the point cloud data from the LiDAR sensor 13 (ST404).

[0099] On the other hand, if the image capturing device 1 is not equipped with a LiDAR sensor 13 (No in ST401), the position and orientation of the color camera at the time of capturing is estimated based on the captured images (frames) at each time included in the captured image data, and position and orientation information for each time regarding the color camera 12 is obtained (ST405).

[0100] Next, the processor 23 performs scale adjustment for the captured images (frames) at each time included in the captured image data based on the position information from the satellite positioning sensor 14 (ST406).

[0101] Next, the processor 23 sets the north direction and the vertical direction for the captured image (frame) at each time included in the captured image data based on the direction information from the IMU sensor 15 (ST407).

[0102] As described above, in this embodiment, when a LiDAR is mounted on the photographing device 1, scale alignment can be performed with sufficient accuracy even with a relatively inexpensive LiDAR sensor, without installing ground control points (GCPs) on the ground. Also, when a satellite positioning sensor 14 such as a GPS is mounted, position alignment can be performed using a world coordinate system based on position information of the photographing location.

[0103] Next, a description will be given of the disaster prevention map data generation process (ST304 in FIG. 5) performed by the disaster prevention map server 3. FIG. 7 is a flowchart showing the procedure of the disaster prevention map data generation process.

[0104] In the disaster prevention map server 3, the processor 33 first reads the area map stored in the memory unit of the device itself (ST501). The area map represents the topography of the target area and serves as the basis for a 2D disaster prevention map that visualizes the risk level of each point. The area map is provided to the disaster prevention map server 3 from the editing terminal 4. The area map may be created by the editing terminal 4 or obtained from an external source.

[0105] Next, processor 33 determines whether the mode is the manual input mode or the data analysis mode (ST502). Note that whether the mode is the manual input mode or the data analysis mode may be set in advance, or may be specified by the user each time.

[0106] Here, if the manual input mode is selected (Yes in ST502), the processor 33 displays the editing screen 61 (see FIGS. 8(B) and 9(A)) on the editing terminal 4, and acquires the risk level of each point (each pixel) on the free viewpoint image 62 in response to a user's operation of inputting the risk level of each point on the editing screen 61 (risk acquisition process) (ST503). At this time, the free viewpoint image 62 from the viewpoint specified by the user is generated using the NeRF model and displayed on the editing screen 61. Also, a point cloud image from the viewpoint specified by the user is generated from the point cloud data and displayed on the editing screen 61.

[0107] On the other hand, if the mode is not manual input mode, i.e., if the mode is data analysis mode (No in ST502), processor 33 generates a free viewpoint image of a specified viewpoint using the NeRF model, performs image recognition processing on the free viewpoint image to determine the risk level of each point (each pixel) on the free viewpoint image, and also performs point cloud recognition processing on the point cloud data to determine the risk level of each point (each pixel) on the free viewpoint image (risk level acquisition processing) (ST504).

[0108] Next, processor 33 sets the risk level of each point on the NeRF mesh data and point cloud data, which are 3D data, based on the risk level of each point (each pixel) on the free viewpoint image, which is 2D data (risk level reflection process) (ST505). Specifically, a risk level is set for each mesh included in the NeRF mesh data. Also, a risk level is set for each point included in the point cloud data.

[0109] Next, processor 33 generates a 2D disaster prevention map in which the risk level of each point is visualized based on the risk level of each point set in the NeRF mesh data and point cloud data, which are 3D data (risk visualization process) (ST506). At this time, a top-view (aerial viewpoint) risk heat map is generated based on the risk level of each point set in the NeRF mesh data and point cloud data, and the risk heat map is superimposed on the area map based on the position information in the world coordinate system, thereby generating a 2D disaster prevention map in which the risk level of each point is visualized.

[0110] 7, processing is performed in either the manual input mode or the data analysis mode, but there is also a configuration in which processing in both the manual input mode and the data analysis mode can be performed. For example, after processing in the data analysis mode is performed first, the user may check the processing results and, if necessary, perform processing in the manual input mode to add to or correct the processing results in the data analysis mode.

[0111] Next, a description will be given of the edit screens 51 and 61 when a new document is created and displayed on the edit terminal 4. Figures 8 and 9 are explanatory diagrams showing the edit screens 51 and 61 when a new document is created.

[0112] In this embodiment, the user can switch between a manual input mode in which the user manually specifies the risk level of each point in the target area, and a data analysis mode in which the risk level of each point in the target area is obtained by data analysis by the processor 33. In the manual input mode, editing screens 51 and 61 are displayed on the editing terminal 4, and the user can input the risk level of each point in the target area on the editing screens 51 and 61.

[0113] Editing screens 51 and 61 are provided with an image display section 52 and an operation section 53. Operation section 53 is provided with a mode display section 54. Mode display section 54 displays a character representing the current mode, i.e., either "2D map" or "3D view."

[0114] First, as shown in Fig. 8(A), an editing screen 51 in 2D map mode is displayed. At this time, an area map 55 is displayed on the image display unit 52. When the user performs an operation to specify a desired point on the area map 55 (for example, by clicking the mouse), the screen transitions to an editing screen 61 in 3D view mode (see Fig. 8(B)).

[0115] As shown in Fig. 8(B), on the editing screen 61 in 3D view mode, a free viewpoint image 62 from an arbitrary viewpoint generated using the NeRF model is displayed on the image display unit 52. This free viewpoint image 62 corresponds to the surrounding area of ​​a point designated by the user on the area map 55 on the editing screen 51 in 2D map mode (see Fig. 8(A)).

[0116] On the editing screen 61 in the 3D view mode, the user can change the direction of the viewpoint by performing a predetermined operation (for example, a mouse drag operation) on the free viewpoint image 62. In addition, the user can switch (enlarge and reduce) the display magnification of the free viewpoint image 62 by performing a predetermined operation (for example, a mouse wheel operation) on the free viewpoint image 62.

[0117] This allows the user to properly determine dangerous locations within the target area by viewing the high-resolution color free viewpoint image 62. Furthermore, the user can observe the local conditions from various directions by changing the viewpoint of the free viewpoint image 62. In this case, when the user determines the risk of a disaster, not only changes in the shape of the local area but also changes in color are important factors. Based on the changes in the shape and color of the local area, the user can check for precursors to landslides, such as exposed ground or cracks, and can determine the level of risk depending on the severity of the precursors.

[0118] Furthermore, in the editing screen 61 in the 3D view mode, the operation unit 53 is provided with an orientation designation unit 63, a depression angle designation unit 64, and a height designation unit 65. The orientation designation unit 63 allows the user to designate the orientation of the viewpoint. Specifically, the user can change the orientation of the viewpoint by performing an operation of moving the pointer 66 (for example, a drag operation with the mouse). The depression angle designation unit 64 allows the user to designate the depression angle of the viewpoint. Specifically, the user can input the depression angle of the viewpoint as a numerical value. The height designation unit 65 allows the user to designate the height of the viewpoint. Specifically, the user can input the height of the viewpoint as a numerical value. This allows the user to designate a viewpoint at any position and direction.

[0119] In addition, initial values ​​are set in advance for the depression angle and height of the viewpoint, and in the initial state of the editing screen 61 in 3D view mode, a free viewpoint image 62 is generated and displayed on the editing screen 61 based on the initial values ​​for the depression angle and height of the viewpoint.

[0120] Furthermore, on the editing screen 61 in 3D view mode, a "top view" button 67 is provided on the operation unit 53. When the user operates the "top view" button 67, a free viewpoint image 62 in which the viewpoint is set directly above the target area is displayed on the image display unit 52. At this time, the depression angle of the viewpoint is set to 90°.

[0121] Furthermore, on the editing screen 61 in the 3D view mode, the user can input the danger level of a specified point by performing an operation to specify a point on the free viewpoint image 62. Specifically, the user can specify the range of a danger area by performing an operation to fill in a required range by dragging the mouse. The user can also specify the range of a danger area by drawing a boundary line surrounding the required range in one stroke by dragging the mouse. The user can also specify the range of a danger area by specifying the vertices of the polygon by clicking the mouse. When a danger area is specified, for example, a dialog box pops up, and the user can input the danger level of the specified danger area. The danger level may be input as a numerical value or as a level (e.g., high, medium, low).

[0122] Furthermore, on the editing screen 61 in the 3D view mode, when the user inputs the danger level, the danger level of each point is displayed in different colors according to the height on the free viewpoint image 62. For example, areas with a high danger level are drawn in red, areas with a medium danger level are drawn in orange, and areas with a low danger level are drawn in yellow.

[0123] The example shown in Fig. 9(A) is an editing screen 61 in 3D view mode, similar to the example shown in Fig. 8(B), but the viewpoint direction is different from that in the example shown in Fig. 8(B). Specifically, the viewpoint direction is set to north in the example shown in Fig. 8(B), whereas the viewpoint direction is set to southeast in the example shown in Fig. 9(A).

[0124] Furthermore, the editing screen 61 in the 3D view mode has a "Back" button 68 on the operation unit 53. When the user operates the "Back" button 68, the screen returns to the editing screen in the 2D map mode, as shown in FIG. 9(B). At this time, unlike the initial state of the 2D map mode (see FIG. 8(A)), a 2D disaster prevention map 69 with a risk heat map superimposed thereon is displayed on the image display unit 52. On the risk heat map, the risk level (high, medium, low) of each point is depicted in different colors.

[0125] In addition to the free viewpoint image 62 generated by the NeRF model, the editing screens 51 and 61 (see Figures 8 and 9) may also display images actually captured by the color camera 12 of the camera 1, particularly images (keyframes) used to train the NeRF model.

[0126] Next, a description will be given of the viewing screens 71 and 81 displayed on the viewing terminal 5. Figures 10 and 11 are explanatory diagrams showing the viewing screens 71 and 81.

[0127] Similar to the editing screens 51 and 61 (see FIGS. 8 and 9), the viewing screens 71 and 81 are provided with an image display unit 52 and an operation unit 53. Also, similar to the editing screens 51 and 61, the viewing screens 71 and 81 have a 2D map mode and a 3D view mode.

[0128] First, as shown in FIG. 10(A), a viewing screen 71 in 2D map mode is displayed. At this time, a 2D disaster prevention map 69 with a risk heat map superimposed thereon is displayed on the image display unit 52. In the risk heat map, the risk level of each point is depicted in a different color according to its level (high, medium, low). When the user performs an operation to specify a desired point on the 2D disaster prevention map 69 (for example, by clicking the mouse), the screen transitions to a viewing screen 81 in 3D view mode (see FIG. 10(B)).

[0129] As shown in Fig. 10(B), on the viewing screen 81 in 3D view mode, a free viewpoint image 62 generated using the NeRF model is displayed on the image display unit 52. This free viewpoint image 62 corresponds to the area surrounding a point designated by the user on the 2D disaster prevention map 69 on the viewing screen 71 in 2D map mode (see Fig. 10(A)). On the free viewpoint image 62, the risk level of each point is displayed in different colors according to its level (high, medium, low).

[0130] On the viewing screen 81 in the 3D view mode, the user can change the direction of the viewpoint by performing a predetermined operation (for example, dragging the mouse) on the free viewpoint image 62. In addition, the user can switch (enlarge and reduce) the display magnification of the free viewpoint image 62 by performing a predetermined operation (for example, operating the mouse wheel) on the free viewpoint image 62. This allows the user to view high-resolution free viewpoint images while changing the viewpoint, thereby allowing them to properly and easily check for dangerous locations within the target area.

[0131] Furthermore, on the viewing screen 81 in the 3D view mode, similar to the editing screen 61 (see FIGS. 8(B) and 9(A)), an orientation designation section 63, a depression angle designation section 64, and a height designation section 65 are provided on the operation section 53. Also, on the viewing screen 81 in the 3D view mode, similar to the editing screen 61, a "top view" button 67 is provided on the operation section 53.

[0132] The example shown in Fig. 11 is a viewing screen 81 in 3D view mode, similar to the example shown in Fig. 10(B), but the viewpoint direction is different from that in the example shown in Fig. 10(B). Specifically, the viewpoint direction is set to north in the example shown in Fig. 10(B), whereas the viewpoint direction is set to southeast in the example shown in Fig. 11.

[0133] Furthermore, on the viewing screen 81 in 3D view mode, similarly to the editing screen 61 (see FIGS. 8(B) and 9(A)), a "back" button 68 is provided on the operation unit 53. When the user operates the "back" button 68, the screen returns to the viewing screen 71 in 2D map mode (see FIG. 10(A)).

[0134] Next, a description will be given of the edit screens 51 and 61 at the time of updating that are displayed on the edit terminal 4. Figures 12 and 13 are explanatory diagrams showing the edit screens 51 and 61 at the time of updating.

[0135] The risk level of each point within the target area changes over time. Therefore, in this embodiment, the user can update the risk level of a desired point within the target area at an appropriate timing using the same procedure as when creating a new area.

[0136] Furthermore, at the timing of updating, the NeRF model is also updated by flying the drone 6 over the target area and having the camera 1 perform 3D sensing (three-dimensional measurement) using the color camera 12 and the LiDAR sensor 13. The updated NeRF model is used on the editing screens 51 and 61 during updating.

[0137] First, as shown in Fig. 12(A), the editing screen 51 in 2D map mode is displayed. At this time, the image display unit 52 displays a 2D disaster prevention map 69 with a risk heat map superimposed on it. This 2D disaster prevention map 69 was created when a new map was created or the last time it was updated. When the user performs an operation to specify a desired point on the 2D disaster prevention map 69, the screen transitions to the editing screen 61 in 3D view mode (see Fig. 12(B)).

[0138] 12(B), on the editing screen 61 in 3D view mode, a free viewpoint image 62 generated using the NeRF model is displayed on the image display unit 52. At this time, based on the risk level set at the time of new creation or the previous update, dangerous areas are displayed in different colors on the free viewpoint image 62 according to the level of risk. On the editing screen 61 in 3D view mode, the user can specify a point on the free viewpoint image 62, thereby correcting and inputting the risk level of the specified point.

[0139] When the user operates the "Back" button 68, the screen returns to the editing screen 51 in 2D map mode, as shown in Fig. 3. At this time, the 2D disaster prevention map 69 with the risk heat map superimposed thereon is displayed on the image display unit 52, reflecting the risk modification operation on the editing screen 61 in 3D view mode.

[0140] In this embodiment, the user can easily update the risk level of required points within the target area, so the 2D disaster prevention map 69 can be kept up to date and reflect the current conditions on site.

[0141] (Modification of the first embodiment) Next, a modified example of the first embodiment will be described. Note that points not particularly mentioned here are the same as those in the above-described embodiment. Fig. 14 is an explanatory diagram showing a vehicle 7 as a moving body on which an imaging device 1 according to a modified example of the first embodiment is mounted.

[0142] In the first embodiment (see FIG. 1), the photographing device 1 is mounted on a drone 6 as a moving object. The drone 6 flies above an area subject to disaster prevention measures, and the photographing device 1 performs 3D sensing (three-dimensional measurement) using a color camera 12 and a LiDAR sensor 13.

[0143] On the other hand, in this modification, the camera 1 is mounted on a vehicle 7 as a moving body. The vehicle 7 travels on roads within an area subject to disaster prevention measures, and the camera 1 performs 3D sensing using a color camera 12 and a LiDAR sensor 13. Note that the vehicle 7 as a moving body is not limited to the four-wheeled automobile shown in the figure, and may also be a motorcycle or a bicycle.

[0144] The vehicle 7 as a moving body may be an infrastructure inspection vehicle. The infrastructure inspection vehicle, for example, patrols locations where communication and power distribution equipment, such as electric wires (power distribution cables, communication cables) and utility poles to be inspected, is installed, and the image capture device 1 performs 3D sensing of the communication and power distribution equipment using the color camera 12 and LiDAR sensor 13. At this time, the area surrounding the communication and power distribution equipment is also included in the target of 3D sensing. Therefore, the image data captured by the color camera 12 and the point cloud data captured by the LiDAR sensor 13 can be used to create disaster prevention maps.

[0145] Furthermore, the vehicle 7 as a moving body may be a general-purpose vehicle equipped with a drive recorder. The drive recorder may be equipped with a color camera 12, and may further be equipped with a satellite positioning sensor 14 and an IMU sensor 15 (see FIG. 2), which may be used as the sensor unit 11 of the image capturing device 1. Furthermore, a general-purpose vehicle may also be equipped with a LiDAR sensor 13, which may be used as the sensor unit 11 of the image capturing device 1.

[0146] In addition, a disaster prevention map is created based on data collected by the camera 1 mounted on the drone 6 (see Figure 1), and data collected by the camera 1 mounted on an infrastructure inspection vehicle or a drive recorder such as a camera 1 mounted on a regular vehicle may be used to supplement the data collected by the camera 1 mounted on the drone 6.

[0147] (Second embodiment) Next, a second embodiment will be described. Note that the points not specifically mentioned here are the same as those in the previous embodiment. Fig. 15 is a diagram showing the overall configuration of a supply chain management system according to the second embodiment.

[0148] When a disaster such as heavy rain or an earthquake occurs, transportation routes such as roads and railways cannot be used as usual, affecting shipping from logistics centers and shipments from production centers. Therefore, it is important to ensure safe transportation routes when a disaster occurs. Therefore, in this embodiment, an optimal delivery route is presented that takes a safe route and avoids dangerous transportation routes that are expected to be obstructed in the event of a disaster. Furthermore, in this embodiment, an optimal delivery route is selected from the perspective of SCM (Supply Chain Management).

[0149] The disaster prevention map generation system of the first embodiment comprises a camera 1, a data processing terminal 2, a disaster prevention map server 3, an editing terminal 4, and a viewing terminal 5, while the supply chain management system of this embodiment comprises, in addition to each device of the disaster prevention map generation system, an SCM server 8 (supply chain management device) and a viewing terminal 9.

[0150] The SCM server 8 utilizes an SCM prediction engine to create plans to optimize each process that makes up the supply chain, such as procurement, manufacturing, logistics, and sales. In this embodiment, as part of SCM (supply chain management), the SCM server 8 predicts the demand for raw materials and products related to warehousing at logistics bases and shipping at production bases, creates a delivery plan, and determines the optimal delivery route based on the delivery plan.

[0151] Viewing terminal 9 is configured as a PC. Viewing terminal 9 is operated by a delivery manager at a logistics or production base. By operating viewing terminal 9 to access SCM server 8, the delivery manager can view a screen displaying the optimal delivery route presented by the SCM prediction engine.

[0152] Next, a description will be given of the general configuration of the SCM server 8. Fig. 16 is a block diagram showing the general configuration of the SCM server 8.

[0153] The SCM server 8 includes a communication unit 91, a storage unit 92, and a processor 93.

[0154] The communication unit 91 communicates with the disaster prevention map server 3 and the viewing terminal 9.

[0155] The memory unit 92 stores programs to be executed by the processor 93 and the like.

[0156] The processor 93 performs various processes by executing programs stored in the storage unit 92. In this embodiment, the processor 93 performs learning processes, delivery route generation processes, and the like.

[0157] In the learning process, the processor 93 constructs an SCM prediction engine by deep learning (machine learning). In this embodiment, the 2D disaster prevention map, which visualizes the risk level and is provided by the disaster prevention map server 3, is used as learning data to construct an SCM prediction engine that takes the 2D disaster prevention map into consideration.

[0158] In the delivery route generation process, the processor 93 uses the SCM prediction engine to generate an optimal delivery route predicted under normal circumstances and an optimal delivery route predicted in the event of a disaster. If the optimal delivery route predicted under normal circumstances has a high potential risk, i.e., if there is a high possibility that traffic will be restricted in the event of a disaster, a delivery route with a low potential risk is generated as the optimal delivery route predicted in the event of a disaster.

[0159] In addition, the SCM server 8 may be provided with an SCM prediction engine for normal times and an SCM prediction engine for disaster situations, and the SCM prediction engine for normal times may be used to generate the optimum delivery route predicted for normal times, and the SCM prediction engine for disaster situations may be used to generate the optimum delivery route predicted for disaster situations.

[0160] Next, a description will be given of viewing screen 101 displayed on viewing terminal 9. FIG.

[0161] The viewing screen 101 is provided with an image display section 102 and a menu section 103. The menu section 103 is provided with a "normal prediction" button 104 and a "prediction taking disaster prevention map into consideration" button 105. The user can switch between the normal prediction mode and the abnormal prediction mode by operating the "normal prediction" button 104 and the "prediction taking disaster prevention map into consideration" button 105.

[0162] 17(A) shows an example of normal prediction mode, that is, a case where the user operates the “normal prediction” button 104. In this case, the optimal delivery route predicted under normal circumstances is displayed on the image display unit 102, superimposed on the area map 106.

[0163] The example shown in Fig. 17(B) is a disaster prediction mode, i.e., a case where the user operates the "Prediction taking disaster prevention map into consideration" button 105. In this case, in addition to the optimal delivery route predicted under normal circumstances, the optimal delivery route predicted in the event of a disaster is superimposed on a 2D disaster prevention map 107 that visualizes the risk level on the image display unit 102. In the risk level heat map, areas with a predetermined risk level (high, medium) are depicted.

[0164] Here, since the optimal delivery route predicted under normal circumstances is likely to be subject to traffic restrictions in the event of a disaster, a different delivery route that is less likely to be subject to traffic restrictions in the event of a disaster is presented as the optimal delivery route predicted in the event of a disaster.

[0165] In the event of a disaster, the delivery manager will not be able to immediately confirm the actual damage situation, but will be able to assume that anticipated damage has occurred and instruct the relevant departments to make deliveries based on the disaster delivery routes.

[0166] 17, a delivery route by land is presented, but the delivery route is not limited to a land route. Optimal delivery routes including air and sea routes in addition to land routes may also be presented.

[0167] As described above, the embodiments have been described as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made. Furthermore, it is also possible to combine the components described in the above embodiments to create new embodiments. [Industrial Applicability]

[0168] The local map generation device, supply chain management system, and local map generation method of the present invention have the effect of being able to appropriately determine the specific status of the site simply by photographing the site, and to generate a highly accurate disaster prevention map that allows the user to appropriately confirm the specific status of the site.They are useful as a local map generation device and local map generation method that generate a local map that visualizes the specific status of the site based on the detection results of a sensor mounted on a mobile object, and as a supply chain management system that presents optimal delivery routes based on supply chain management. [Explanation of symbols]

[0169] 1: Camera 2: Data processing terminal 3: Disaster prevention map server (local map generator) 4: Editing terminal (display device) 5: Viewing terminal (display device) 6. Drone 7: Vehicle 8: SCM server 9: Viewing terminal (display device) 11: Sensor section 12: Color camera 13: LiDAR sensor 23: Processor 33: Processor 51: 2D map mode editing screen 55: Area Map 61: Editing screen in 3D view mode 62: Free viewpoint image 63: Direction designation part 64: Depression angle specification part 65: Height specification section 69: 2D disaster prevention map 71: 2D map mode viewing screen 81: Viewing screen in 3D view mode 93: Processor 101: Viewing screen 106: Area Map 107: 2D Disaster Prevention Map

Claims

1. A local map generating device including a processor that performs processing to generate a local map that visualizes a specific state of a local area based on detection results of a sensor mounted on a moving object, The processor: Acquiring photographed image data as a result of detection by the visible light camera serving as the sensor; A free viewpoint image is generated based on the captured image data, and information on a specific state of the site is obtained from the free viewpoint image. The local map generating device generates a local map that visualizes the specific state of the site based on information about the specific state.

2. The processor: acquiring point cloud data as a detection result of a three-dimensional sensor as the sensor; 2. The local map generating device according to claim 1, wherein the scale of the photographed image data is adjusted based on the point cloud data.

3. The processor: The local map generating device according to claim 1, characterized in that it uses an image recognition model constructed by machine learning to determine a specific state of the local area from the free viewpoint image taken from a viewpoint in a predetermined direction, and obtains information about the specific state.

4. The processor: displaying an editing screen including the free viewpoint image on a display device; 2. The local map generating device according to claim 1, wherein information about a specific state of a local area is acquired in response to a user's operation of inputting the specific state of the local area for the free viewpoint image on the editing screen.

5. The processor: generating the free viewpoint image from a designated viewpoint in response to a user operation specifying an arbitrary viewpoint; 2. The local map generating device according to claim 1, wherein a viewing screen including the free viewpoint image from a specified viewpoint is displayed on a display device.

6. The processor: generating the free viewpoint image based on pre-stored initial values ​​related to the depression angle and height of the viewpoint and displaying the free viewpoint image on the display device; 5. The local map generating device according to claim 4, wherein the free viewpoint image is generated from a viewpoint having a specified depression angle and height in response to a user operation specifying the depression angle and height of the viewpoint, and is displayed on the display device.

7. The on-site map generating device according to claim 1; a supply chain management device that presents an optimal delivery route based on supply chain management; The supply chain management device includes: acquiring the local map that visualizes a specific state of the local area from the local map generating device; A supply chain management prediction engine is constructed using machine learning with the local map as learning data, This supply chain management system uses the supply chain management prediction engine to generate optimal delivery routes predicted under normal circumstances and optimal delivery routes predicted in the event of a disaster.

8. A local map generation method that causes a processor to perform a process of generating a local map that visualizes a specific state of a local area based on a detection result of a sensor mounted on a moving object, the method comprising: Acquiring photographed image data as a result of detection by the visible light camera serving as the sensor; A free viewpoint image is generated based on the captured image data, and information on a specific state of the site is obtained from the free viewpoint image. A local map generating method for generating a local map that visualizes the specific state of the site based on information about the specific state.

Images