Program, information processing method, and information processing apparatus

Abstract

We provide a program that can identify the location of trees that obstruct vehicle movement. [Solution] A program according to one embodiment of the present disclosure acquires an image taken by an in-vehicle camera and location information indicating the location where the image was taken. When the acquired image is input, the program inputs the image to a learning model that outputs the range in the image that contains an obstruction tree that would hinder vehicle driving, outputs the range that contains the obstruction tree, and causes a computer to perform a process of associating the image containing the obstruction tree with the location information of the image containing the obstruction tree and storing it.

Description

Technical Field

[0001] This technology relates to programs, information processing methods, and information processing apparatuses.

Background Art

[0002] Conventionally, a system for discriminating the quality of trees has been proposed. For example, the tree discrimination system of Patent Document 1 has an information acquisition means for acquiring aerial image information obtained by imaging a forest for discriminating the quality of trees from the air, and a discrimination means for discriminating the quality of trees based on reference aerial image information obtained by imaging the forest from the air in the past.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The technology disclosed in Patent Document 1 does not consider grasping the position information of obstacle trees that hinder vehicle travel.

[0005] In view of such circumstances, the present disclosure has been made, and an object thereof is to provide a program or the like capable of grasping the position information of obstacle trees that hinder vehicle travel.

Means for Solving the Problems

[0006] A program according to one embodiment of the present disclosure acquires an image captured by an in-vehicle camera and location information indicating the location where the image was captured. When the acquired image is input, the program inputs the image to a learning model that outputs the range in the image that contains an obstruction tree that would hinder vehicle driving, outputs the range containing the obstruction tree, and causes a computer to perform a process of associating the image containing the obstruction tree with the location information of the image containing the obstruction tree and storing it. [Effects of the Invention]

[0007] In one embodiment of the present disclosure, the program can determine the location of obstructing trees that hinder vehicle movement. [Brief explanation of the drawing]

[0008] [Figure 1] This is an explanatory diagram showing an example of a system for identifying problematic trees. [Figure 2] This is a block diagram showing an example of an information processing device. [Figure 3] This is a block diagram showing an example configuration of a terminal device. [Figure 4] This is an explanatory diagram showing an example of a problematic tree detection model. [Figure 5] This is an explanatory diagram showing an example of a domain partitioning model. [Figure 6] This is an explanatory diagram showing an example of a distance calculation model. [Figure 7] This is an explanatory diagram showing an example of a failure risk assessment module. [Figure 8] This is an explanatory diagram showing an example of a distance marker database. [Figure 9] This is an explanatory diagram showing an example of a compatible database. [Figure 10] This is an explanatory diagram showing an example of a driving image database. [Figure 11] This is an explanatory diagram showing an example of a driving image transmission screen. [Figure 12] This is an explanatory diagram showing an example of a screen displaying the location of obstructing trees. [Figure 13] This is an explanatory diagram showing an example of a driving image display screen. [Figure 14] This is an explanatory diagram showing an example of a correction request screen. [Figure 15] This flowchart shows an example of the process for registering obstructive trees. [Figure 16] This flowchart shows an example of the process of displaying and correcting driving images. [Figure 17] This flowchart shows an example of the retraining process for a problem tree detection model. [Modes for carrying out the invention]

[0009] (Embodiment) Figure 1 is an explanatory diagram showing an example of a hazardous tree detection system S. The hazardous tree detection system S includes an information processing device 1 and a terminal device 2. The terminal device 2 is a terminal used by the user and acquires driving images taken by an on-board camera 3 mounted on the vehicle, for example by transferring them via a recording medium R, and transmits the driving images and location information indicating the location where the driving images were taken to the information processing device 1. The information processing device 1 is, for example, a server device and stores information about hazardous trees based on the driving images acquired from the terminal device 2 and transmits the stored information about hazardous trees to the terminal device 2. Hazardous trees include, for example, dead trees with all leaves withered and a high risk of falling, partially dead trees where the leaves at the tips of the branches are withered but the part closer to the trunk is not withered and is in the process of becoming dead, and overhanging trees whose branches and leaves extend onto the road. Information about hazardous trees includes, for example, location information of the place where the driving image containing the hazardous tree was taken, the determination result of whether or not the tree included in the driving image is a hazardous tree, or information related to the correction of the determination result. The information processing device 1 may output information regarding obstructing trees to terminal devices other than terminal device 2, or to the information processing device 1. The driving images are not limited to those taken while the vehicle equipped with the on-board camera 3 is in motion, but also include images taken while the vehicle is stopped. In this embodiment, the recording medium R is, for example, an SD card, but the recording medium R may be a CD-ROM, USB (Universal Serial Bus) memory, microSD card, or CompactFlash memory (registered trademark), etc. The terminal device 2 may acquire driving images from the on-board camera 3 via internet communication. The information processing device 1 may also acquire driving images from terminal device 2 or the on-board camera 3 via internet communication.

[0010] Figure 2 is a block diagram showing an example of an information processing device 1. The information processing device 1 comprises a control unit 11, a storage unit 12, and a communication unit 13. Each component is connected by a bus B. The control unit 11 includes arithmetic processing units such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), and a GPU (Graphics Processing Unit), and performs various information processing and control processing related to the information processing device 1 by reading and executing a program P (program product) stored in the storage unit 12. The program P can be deployed on a single computer, at a single site, or distributed across multiple sites and executed on multiple computers interconnected by a communication network. The information processing device 1 may also be a device corresponding to a node on a blockchain. In Figure 1, the control unit 11 is described as a single processor, but it may be a multiprocessor.

[0011] The storage unit 12 of the information processing apparatus 1 is, for example, a volatile memory and a non-volatile memory. The storage unit 12 stores a program P, a fault tree detection model (learning model) M1, a region division model M2, a distance calculation model M3, a hazard risk evaluation module M4, a distance standard DB (database) 121, a correspondence DB 122, and a driving image DB 123. Note that the program P may be provided to the information processing apparatus 1 using a recording medium 12a that stores the program P in a computer-readable manner. The recording medium 12a is, for example, a portable memory. Examples of the portable memory include a CD-ROM, a USB memory, an SD card, a micro SD card, or a compact flash memory (registered trademark). When the recording medium 12a is a portable memory, the processing element of the control unit 11 may read the program P from the recording medium 12a using a reading device (not shown). The read program P is written into the storage unit 12. Further, the program P may be provided to the information processing apparatus 1 when the communication unit 13 communicates with an external device. Details of the fault tree detection model M1, the region division model M2, the distance calculation model M3, the hazard risk evaluation module M4, the distance standard DB (database) 121, the correspondence DB 122, and the driving image DB 123 will be described later.

[0012] The communication unit 13 of the information processing apparatus 1 is a communication module or a communication interface for communicating with other devices such as the terminal device 2 by wire or wirelessly, and is, for example, a wide area wireless communication module such as LTE (registered trademark), 4G, or 5G. The control unit 11 communicates with the terminal device 2 through an external network N such as the Internet via the communication unit 13.

[0013] FIG. 3 is a block diagram showing a configuration example of the terminal device 2. The terminal device 2 includes a terminal control unit 21, a storage unit 22, a communication unit 23, a display unit 24, and an input unit 25. The terminal control unit 21 is composed of a CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphics Processing Unit), NPU (Neural Processing Unit), or TPU (Tensor Processing Unit), etc., and performs various control processes, arithmetic processes, etc. Note that the functions of the terminal device 2 may be realized by a plurality of devices, or may be a device corresponding to a node on the blockchain. Also, the terminal device 2 may perform part or all of the processes executed by the information processing device 1.

[0014] The storage unit 22 of the terminal device 2 stores an application program (app program) Pa of an application that transmits a driving image acquired from the in-vehicle camera 3 to the information processing device 1 and displays information on obstacles received from the information processing device 1. The app program Pa is provided to the terminal device 2 using, for example, a recording medium 22a. The recording medium 22a is, for example, a portable memory. Examples of the portable memory include a CD-ROM, a USB (Universal Serial Bus) memory, an SD card, a micro SD card, or a compact flash memory (registered trademark), etc. When the recording medium 22a is a portable memory, the processing element of the terminal control unit 21 may read the app program Pa from the recording medium 22a by a reading unit 26. Note that the terminal control unit 21 of the terminal device 2 may acquire the app program Pa using the Internet and store it in the storage unit 22. Note that the terminal device 2 and the in-vehicle camera 3 may be integrally configured. The in-vehicle camera 3 may be realized by the functions of the terminal device 2.

[0015] The communication unit 23 of the terminal device 2 is a communication module or communication interface for communicating with the information processing device 1 via wired or wireless means. The terminal control unit 21 communicates with the information processing device 1 via the communication unit 23 and through an external network N.

[0016] The display unit 24 of the terminal device 2 is composed of a display and displays a screen for transmitting driving images acquired from the in-vehicle camera 3 to the information processing device 1, or a screen for displaying information about obstructing trees acquired from the information processing device 1 and for accepting input for corrections, etc.

[0017] The input unit 25 of the terminal device 2 is configured, for example, with a keyboard or mouse, and accepts input of various information or modifications. If the terminal device 2 is a tablet terminal, the display unit 24 and the input unit 25 may be configured as a single unit with a touch panel.

[0018] The reading unit 26 is, for example, a reader capable of reading data from the recording medium R. The terminal control unit 21 acquires driving images captured by the in-vehicle camera 3 from the recording medium R via the reading unit 26. Alternatively, the terminal control unit 21 may acquire driving images by communicating with the in-vehicle camera 3 via the communication unit 23, using wireless communication including short-range wireless communication or internet communication, or wired communication.

[0019] Figure 4 is an explanatory diagram showing an example of the obstructive tree detection model M1. The obstructive tree detection model M1 is a model with object detection capabilities such as CNN (Convolutional Neural Network), RCNN (Regions with Convolutional Neural Network), Fast RCNN, Faster RCNN, SSD (Single Shot Multibook Detector), YOLO (You Only Look Once), or Vision Transformer. The obstructive tree detection model M1 is trained to output the range, type, and species of obstructive trees when a driving image is input, using training data that has been pre-associated with driving images and the range, type, and species of obstructive trees contained in the driving images.

[0020] If the obstruction tree detection model M1 is composed of a neural network including a CNN that extracts image features, such as RCNN, the obstruction tree detection model M1 has multiple neurons that accept pixel values ​​from the driving image as input and passes the input pixel values ​​to the hidden layer. The hidden layer has multiple neurons that extract image features from the driving image and passes the extracted image features to the output layer. Based on the image features, the output layer outputs the range in the driving image that contains the obstruction tree, the type of obstruction tree (dead tree, partially dead tree, or overhanging tree), and the tree species of the obstruction tree. The control unit 11 may also output the location and type of the obstruction tree using a method such as pattern matching. The obstruction tree detection model M1 may also output "normal tree or not applicable" as the type of obstruction tree. The obstruction tree detection model M1 may also output the range of the obstruction tree by segmentation. The obstruction tree detection model M1 may determine three or more types of obstruction trees.

[0021] Figure 5 is an explanatory diagram showing an example of a region segmentation model M2. The region segmentation model M2 is a model that divides an image into multiple regions, possessing segmentation functions such as RCNN, Fast RCNN, Faster RCNN, SSD, YOLO, or Vision Transformer.

[0022] The domain segmentation model M2 is trained to output images segmented in the same way as the annotation images when a driving image is input, using training data that associates driving images with images that have been pre-annotated into three regions: road region, planted region, and empty region (annotated images). Alternatively, the domain segmentation model M2 may output images that have been segmented into two regions: road region and non-road region.

[0023] If the region segmentation model M2 is composed of a neural network including a CNN that extracts image features, such as RCNN, the region segmentation model M2 has multiple neurons that accept pixel values ​​from the driving image as input and passes the input pixel values ​​to the hidden layer. The hidden layer has multiple neurons that extract image features from the driving image and passes the extracted image features to the output layer. The output layer outputs the boundaries of the road region, planted region, and empty region based on the image features. In the example shown in the output image (region segmentation image) of Figure 5, the boundaries of the road region, planted region, and empty region are each indicated by different hatching. The region segmentation model M2 may also output identification information indicating which region each pixel constituting the driving image belongs to.

[0024] Figure 6 is an explanatory diagram showing an example of the distance calculation model M3. The distance calculation model M3 is based on a markerless AR method, such as SLAM (Simultaneous Localization and Mapping) or SmartAR. The distance calculation model M3 is trained using pre-trained data that associates images containing objects captured by the in-vehicle camera 3 with the actual dimensions of the objects and the measured distance from the in-vehicle camera 3 to the plane on which the objects are located. When an image is input, the model is trained to output the dimensions of the objects contained in the image, the distance from the in-vehicle camera 3 to the plane on which the objects are located, and the distance from the base of the obstructing tree to the road area.

[0025] When a driving image including an obstructing tree is input, the distance calculation model M3 outputs the actual size (tree height) of the obstructing tree, the distance from the on-board camera 3 to the plane containing the obstructing tree (obstructing tree plane) (obstructing tree plane distance), and the distance from the base of the obstructing tree to the road area (separation distance). The separation distance is output based on the horizontal length in the image from the center of the bottom edge of the rectangle indicating the extent of the obstructing tree to the road area. The tree height of the obstructing tree, the distance from the on-board camera 3 to the plane where the obstructing tree exists, and the separation distance may be measured by functions such as Lidar (Light Detection And Ranging) or infrared sensors of the vehicle on which the on-board camera 3 is mounted, or they may be measured based on PTAM (Parallel Tracking and Mapping). If the obstructing tree is an overhanging tree, the distance calculation model M3 may output the length of the branches extending approximately horizontally in the driving image.

[0026] Figure 7 is an explanatory diagram showing an example of the obstacle risk assessment module M4. The obstacle risk assessment module M4 is a function of program P and is executed by the control unit 11. When the obstructing tree is a dead tree, the control unit 11 of the information processing device 1 transforms the range of the obstructing tree in the driving image based on the tree height, the planar distance to the obstructing tree, and the separation distance output by the distance calculation model M3, and generates an image of what would happen if the obstructing tree fell. The image is generated by superimposing an image of the range of the obstructing tree rotated 90 degrees towards the region side, centered on the midpoint of the lower edge of the range. Based on the image and the region division image output by the region division model M2, the control unit 11 calculates the obstacle risk to the road area when the obstructing tree falls (the ratio of the length of the fallen obstructing tree to the width of the road area in the region division image). The control unit 11 evaluates the obstacle risk by classifying it into three stages, for example, "low," "medium," and "high," according to the degree of obstacle risk, and stores the classified obstacle risk evaluation in the storage unit 12 in association with the driving image. Furthermore, if the obstructing tree is an overhanging tree, the control unit 11 may calculate the risk of the obstructing tree's branches interfering with the road area based on the driving image, the region division image output by the region division model M2, and the branch length output by the distance calculation model M3.

[0027] Figure 8 is an explanatory diagram showing an example of a distance marker DB121. The distance marker DB121 stores location information of distance markers (kilometer posts or meter posts) installed on roads. The management items (fields) of the distance marker DB include a road number field, a distance marker field, a north latitude field, and an east longitude field. The road number field stores the road number of the road on which the distance marker is installed. The distance marker field stores the distance (km) from the road starting point indicated by the distance marker. The north latitude field stores the north latitude of the distance marker's location. The east longitude field stores the east longitude of the distance marker's location. The distance marker DB may also store the south latitude or west longitude of the distance marker. Furthermore, the distance marker DB121 may consist of multiple DBs, one for each road, each having the road number as attribute information. In this embodiment, north latitude and east longitude are shown in decimal degrees, but they may also be shown in degrees, minutes, and seconds.

[0028] Figure 9 is an explanatory diagram showing an example of the response DB122. The response DB122 stores the required response level according to the type of tree in the problem and the damage risk assessment. The management items (fields) of the response DB122 include the tree type field, the damage risk assessment field, and the required response level field. The tree type field stores the type of tree in the problem (dead tree, partially dead tree, or overhanging tree). The damage risk assessment field stores the damage risk assessment output by the damage risk assessment module M4, which is based on the tree's fall or overhanging branches. The required response level stores the required response level according to the tree type and the damage risk assessment (for example, a 5-level value with a maximum of 5).

[0029] Figure 10 is an explanatory diagram showing an example of a driving image DB 123. Terminal device 2 transmits a series of frame images (driving images) captured by the in-vehicle camera 3 as a video to information processing device 1. The control unit 11 of the information processing device 1 stores information about the acquired series of driving images in the driving image DB 123 of the storage unit 12. The control unit 11 stores information about the driving images in the driving image DB 123 for each predetermined frame (for example, every 10 frames) from the acquired series of frame images. The management items of the driving image DB 123 include a shooting date and time field, a driving image field, a shooting location latitude field, a shooting location longitude field, a nearest distance marker field, an upper / lower classification field, a number of obstructing trees field, an obstructing tree range field, an obstructing tree type field, an obstructing tree species field, an obstructing tree height field, a separation distance field, an image field, an obstacle risk assessment field, a response necessity field, an obstructing tree range correction field, an obstructing tree judgment correction field, a tree species correction field, a tree height correction field, a separation distance correction field, and a remarks field. The storage unit 12 of the information processing device 1 may store information about the vehicle's movement or obstructing trees as an array of structured data.

[0030] The date and time field in the driving image DB123 stores the date and time the driving image was taken. The driving image field stores the taken driving image, for example, in file format. The latitude field of the shooting location stores the latitude of the location where the driving image was taken. The longitude field of the shooting location stores the longitude of the location where the driving image was taken. The nearest distance marker field stores the latitude and longitude of the shooting location, as well as the road number and distance from the road's starting point of the nearest distance marker, which is identified based on the distance marker DB121. If there is no distance marker whose location is stored in the distance marker DB121 within a predetermined distance (for example, a radius of 50m) from the location where the driving image was taken, the nearest distance marker field will be left blank. In this case, there will be no nearest distance marker for that driving image. The presence or absence of a nearest distance marker indicates the reliability of the location information. The Up / Down Classification field stores the classification of the vehicle's direction of travel (up or down), which is determined based on the latitude and longitude of the shooting location, or the transition of latitude and longitude for each frame, on which the vehicle camera 3 is mounted.

[0031] The number of obstructive trees field in the driving image DB123 stores the number of obstructive trees in the driving image output by the obstructive tree detection model M1. The obstructive tree range field stores the normalized pixel coordinates of the center of the range containing the obstructive trees in the driving image, output by the obstructive tree detection model M1, where the upper left corner of the driving image is (0,0) and the lower right corner is (1,1). In this embodiment, the range containing the obstructive trees is indicated by a rectangular range, and the obstructive tree range field stores the pixel coordinates, width, and height of the center of the range containing the obstructive trees. The obstructive tree type field stores the type of obstructive tree output by the obstructive tree detection model M1. If the driving image contains multiple obstructive trees, the obstructive tree type field stores the type of each obstructive tree. The obstructive tree species field stores the species of the obstructive trees output by the obstructive tree detection model M1. If the driving image contains multiple obstructive trees, the obstructive tree species field stores the species of each obstructive tree. The "Obstructive Tree Height" field stores the height of the obstructive tree output by the distance calculation model M3. If the driving image contains multiple obstructive trees, the "Obstructive Tree Height" field may store multiple values, or it may store the height of the tallest obstructive tree. The "Separation Distance" field stores the separation distance between the obstructive tree and the road area, output by the distance calculation model M3.

[0032] The image field in the driving image DB123 stores the image generated by the fault risk assessment module M4, for example, in file format. An image is stored in the image field only if the obstructing tree is a dead or partially dead tree and an image is generated. If the obstructing tree is anything other than a dead or partially dead tree (e.g., an overhanging tree), no image is generated, and the image field is not stored. The fault risk assessment field stores the fault risk assessment output by the fault risk assessment module M4 regarding the risk of obstructing trees falling or the risk of obstructing branches.

[0033] The "Need for Action" field in the driving image DB123 stores the level of need for action, which is determined by referring to the response DB based on the type of obstacle tree and the fault risk assessment.

[0034] The Obstructive Tree Range Correction field in the driving image DB123 stores the pixel coordinates, width, and height of the center of the range containing the obstructive tree corrected by terminal device 2. The Obstructive Tree Judgment Correction field stores the type of obstructive tree corrected by terminal device 2. If a correction input is made indicating that the obstructive tree detected by the Obstructive Tree Detection Model M1 is not actually an obstructive tree, the Obstructive Tree Judgment Correction field stores "Normal tree or not applicable". In other words, if a correction input is made indicating that it is not an obstructive tree, the information processing device 1 is fed back that the obstructive tree is a normal tree or not subject to type determination. If no correction to the range or type of obstructive tree is input in terminal device 2, an empty value is stored in the Obstructive Tree Range Correction field or the Obstructive Tree Judgment Correction field. The Tree Species Correction field stores the tree species of the obstructive tree corrected by terminal device 2. The Tree Height Correction field stores the tree height corrected by terminal device 2. The Separation Distance Correction field stores the separation distance corrected by terminal device 2. The remarks field stores the remarks entered on terminal device 2.

[0035] Figure 11 is an explanatory diagram showing an example of a driving image transmission screen. Before starting to transmit a series of driving images to the information processing device 1, the terminal control unit 21 of the terminal device 2 displays the driving image transmission screen on the display unit 24. The driving image transmission screen displays a driving road selection field and a file selection field.

[0036] The road selection field on the driving image transmission screen displays the names of multiple roads on which the vehicle is traveling and where driving images are taken. Of the road names, the selected road name is displayed in a dark color, while the other road names are displayed in a light color. The terminal control unit 21 may accept the selection of a road number, or it may accept the input of either a road name or a road number. The file selection field displays the file names of consecutive driving images (videos) read from the recording medium R by the reading unit 26 of the terminal device 2. Each file name is accompanied by a toggle button. The user can select a file to send to the information processing device 1 by turning on the toggle button attached to any of the file names. In addition, the road selection field may automatically select road names based on the location information of where the driving images were taken. Furthermore, if the driving video spans multiple roads, multiple road names may be selected.

[0037] Additionally, a send button is displayed on the driving image transmission screen. When the send button is pressed, the terminal control unit 21 of the terminal device 2 transmits the selected road name and the continuous driving images (video) related to the file to the information processing device 1.

[0038] Figure 12 is an explanatory diagram showing an example of the obstruction tree location display screen. The terminal control unit 21 of the terminal device 2 acquires information about obstruction trees from the information processing device 1. Based on the information about obstruction trees acquired by the terminal control unit 21, the location where the driving image containing the obstruction tree was taken (the location of the obstruction tree) is displayed on the map on the obstruction tree location display screen. The location of the obstruction tree is indicated by a pin mark. The pin mark indicating the location of the obstruction tree differs depending on the type of obstruction tree included in the driving image, whether or not there are corrected items, or whether or not there are nearest distance markers. In other words, the method of displaying the location of the obstruction tree differs depending on the type of obstruction tree included in the driving image, whether or not there are corrected items, or whether or not there are nearest distance markers. Note that corrected items may include items in which the user has entered that the content stored in the driving image DB 123 is correct. In other words, corrected items are items in which the user has provided feedback on the correctness of the content.

[0039] Inside the pin mark, a mark is displayed according to the type of obstructing tree and the presence or absence of the nearest distance marker. In this embodiment, if there is a nearest distance marker for the obstructing tree and the obstructing tree is a dead tree, an "x" mark is displayed inside the pin mark. If the obstructing tree is a partially dead tree, a "▲" mark is displayed inside the pin mark. If the obstructing tree is an overhanging tree, a "-" mark is displayed inside the pin mark. If there is no nearest distance marker, an "*" mark is displayed inside the pin mark. Note that the mark inside the pin mark may differ depending on whether there are normal trees in the driving image or if the image is not relevant. Furthermore, the methods of displaying the inside of the pin mark are not limited to those described above.

[0040] Furthermore, the color of the pin mark varies depending on whether there are corrected items or whether there are nearest distance markers. In this embodiment, if there are corrected items, the pin mark is dark. If there are no corrected items, the pin mark is light. If there are no nearest distance markers, the pin mark is medium-dark. Also, if the upper / lower section is uphill, the sharp part of the pin mark is displayed facing downwards. If the upper / lower section is downhill, the sharp part of the pin mark is displayed facing upwards. Note that the display method of the pin mark may differ based on the number of obstructing trees included in the driving image and the degree to which the obstructing trees need to be addressed. Also, the methods for differentiating the display method of the locations where obstructing trees exist are not limited to those described above. Pin marks selected by the user are surrounded by a thick line (see Figure 13). Note that the methods for differentiating pin marks selected by the user are not limited to those described above.

[0041] Figure 13 is an explanatory diagram showing an example of a driving image display screen. When a pin mark is selected on the obstruction tree position display screen shown in Figure 12, the terminal control unit 21 of the terminal device 2 acquires the driving image and information about the obstruction tree related to the selected pin mark from the information processing device 1, and displays the acquired information on the display unit 24 on the driving image display screen.

[0042] The driving image display screen displays a map showing the location where the driving image was taken (the location where the obstructing tree exists), the date and time of shooting read from the driving image DB123, the road name corresponding to the driving road number, the nearest distance marker, the upper / lower classification, the type of obstructing tree, the species of the obstructing tree, the height of the obstructing tree, the separation distance, the assessment of the obstacle risk, the degree of need for action, and the driving image. The driving image display screen may also display an image of what would happen if the obstructing tree fell, generated by the obstacle risk assessment module M4, as shown in Figure 13. The driving image display screen may accept the selection of either the image or the driving image from several frames before or after the displayed image (e.g., 5 frames), and the selected frame may be displayed. A judgment correction button is displayed on the driving image display screen. When the judgment correction button is pressed, the terminal control unit 21 displays the judgment correction screen (see Figure 14) on the display unit 24.

[0043] Figure 14 is an explanatory diagram showing an example of a correction request screen. The correction request screen displays a map showing the location where the driving image was taken, as well as the driving image itself. Users can make corrections by inputting the range in the driving image on the correction request screen that includes the obstructing tree. In Figure 14, the range output by the obstructing tree detection model M1 is shown by a solid line, and the input (corrected) range is shown by a dashed line.

[0044] Furthermore, users can input corrections to the types of obstructing trees included in the driving images on the correction request screen. The correction request screen displays a field for correcting the type of obstructing tree, each containing a selection of obstructing tree types with a toggle button. The selection of obstructing tree types includes "normal tree or not applicable," "dead tree," "partially dead tree," and "overhanging tree." When the correction request screen is displayed, the toggle buttons for the obstructing tree type selections output by the obstructing tree detection model M1 are turned on. Users can turn on any of the toggle buttons for each selection. The terminal control unit 21 may also accept remarks comments on the correction request screen.

[0045] The correction request screen displays a button to send corrections. When the button to send corrections is pressed, the terminal control unit 21 of terminal device 2 sends the range containing the obstructing tree within the corrected driving image and the type of obstructing tree to the information processing device 1. Corrections to the obstructing tree information, including tree species, tree height, separation distance, route name, facility name, upper / lower classification, or nearest distance marker, may also be entered on the correction request screen. Furthermore, remarks may be entered on the correction request screen. The control unit 11 of information processing device 1 stores the pixel coordinates, width, and height of the center of the corrected range in the obstructing tree range correction field of the driving image DB 123, and stores the type of the corrected obstructing tree in the obstructing tree determination correction field. When a value is stored in the obstructing tree range correction field or the obstructing tree determination correction field of the driving image DB 123, the range or type of the obstructing tree and the corresponding driving image are used as training data for retraining the obstructing tree detection model M1.

[0046] Figure 15 is a flowchart showing an example of the obstruction tree registration process. The control unit 11 of the information processing device 1 acquires a driving image and location information of the location where the driving image was taken from the terminal device 2 (S1). The control unit 11 inputs the driving image to the obstruction tree detection model M1 (S2) and outputs the range, type, and species of obstruction trees included in the driving image (S3). The control unit 11 inputs the driving image to the region division model M2 (S4) and outputs the region division image (S5). The control unit 11 inputs the driving image in which an obstruction tree has been detected to the distance calculation model M3 (S6) and outputs the tree height, separation distance of the obstruction tree, and the distance from the on-board camera 3 to the plane containing the obstruction tree (obstruction tree plane distance) (S7).

[0047] The control unit 11 of the information processing device 1 inputs the driving image, area segmentation image, area of ​​the obstructing tree, height of the obstructing tree, separation distance, and planar distance of the obstructing tree to the obstructing tree (S8), and outputs an image of what would happen if the obstructing tree fell, the risk of damage to the road area if the obstructing tree fell, and a damage risk assessment based on the risk of damage (S9).

[0048] The control unit 11 of the information processing device 1 refers to the distance marker DB 121 and identifies the nearest distance marker and upper / lower classification based on the location information of the location where the driving image was taken (S10). The control unit 11 refers to the corresponding DB 122 and identifies the degree of need for action based on the type of obstructing tree and the obstacle risk assessment (S11). The control unit 11 stores information related to the driving image (date and time the driving image was taken, driving image, latitude of the shooting location, longitude of the shooting location, nearest distance marker, and upper / lower classification) and information related to obstructing trees (number of obstructing trees included in the driving image, range of obstructing trees, type of obstructing tree, species of obstructing tree, height of obstructing tree, separation distance, obstacle risk assessment, and degree of need for action) in the driving image DB 123 (S12), and terminates processing. Note that the driving image does not necessarily have to be input to the distance calculation model M3, and the obstacle risk may be calculated based on the range of obstructing trees in the driving image.

[0049] Figure 16 is a flowchart showing an example of the driving image display and correction process. The terminal control unit 21 of the terminal device 2 transmits information to the information processing device 1 requesting information about the driving image and obstructing trees (S21). The control unit 11 of the information processing device 1 reads information about the driving image and obstructing trees related to multiple driving images from the driving image DB 123 (S22), and transmits the read information about the driving image and obstructing trees to the terminal device 2 (S23).

[0050] The terminal control unit 21 of the terminal device 2 displays an obstruction tree location display screen, which shows different pin marks depending on the type of obstruction tree, whether or not there are corrected items, or whether or not there is a nearest distance marker, based on the information received from the information processing device 1 (S24). The terminal control unit 21 accepts the selection of one of the pin marks on the obstruction tree location display screen (S25).

[0051] The terminal control unit 21 of terminal device 2 displays a driving image display screen that shows the driving image and information about obstructing trees (S26). If the correction button is pressed on the driving image display screen, the terminal control unit 21 displays a judgment correction screen (S27). The terminal control unit 21 accepts input for correction of the range, type, species, height, or distance from the obstructing trees in the driving image (S28). The terminal control unit 21 may also accept input or correction of information about the obstructing trees, including the name of the management office, route name, facility name, upper / lower section, or nearest distance marker. The terminal control unit 21 transmits the range, type, species, height, or distance from the obstructing trees for which correction has been input to the information processing device 1 (S29) and corrects the processing.

[0052] The control unit 11 of the information processing device 1 stores the corrected range, type, species, height, or distance from the obstructing tree received from the terminal device 2 in the driving image DB 123 (S30), and then terminates the process.

[0053] Figure 17 is a flowchart showing an example of the retraining process for the obstruction tree detection model M1. The control unit 11 of the information processing device 1 reads the driving image stored in the driving image field, the corrected range stored in the obstruction tree range correction field, the corrected obstruction tree type stored in the obstruction tree judgment correction field, and the corrected obstruction tree species stored in the tree species correction field from the driving image DB 123 (S41). The control unit 11 creates training data by associating the information read in S41 (S42). The control unit 11 inputs the training data to the obstruction tree detection model M1 (S43), retrains the obstruction tree detection model M1 based on the training data (S44), and terminates the process. The obstruction tree detection model M1 may be retrained based on driving images captured by multiple in-vehicle cameras 3. The obstruction tree detection model M1 may also be retrained when a certain number of corrected obstruction tree ranges or types are stored in the driving image DB.

[0054] With the above configuration and processing, it is possible to detect obstructing trees that hinder vehicle movement and to grasp the location information of the obstructing trees based on driving images captured by the on-board camera 3 mounted on the vehicle. In this embodiment, the range and type of obstructing trees in the driving image are output by the obstructing tree detection model M1 and stored in the driving image DB123, but they may also be manually input by the terminal device 2 and stored in the driving image DB123. Furthermore, the control unit 11 of the information processing device 1 may identify the nearest distance marker to the captured location by recognizing the distance markers included in the driving image using an image recognition model. In addition, the display method of the pin marks displayed on the obstructing tree location display screen may differ depending on whether or not the range or type of obstructing trees has been modified. Part or all of the driving image DB123 may be transmitted to the terminal device 2 and displayed on the display unit 24 of the terminal device 2. In this embodiment, the on-board camera 3 is mounted on a vehicle traveling on a road and photographs the road, but the on-board camera 3 may also be mounted on a railway vehicle and photograph the tracks (railway). The road photographed by the in-vehicle camera 3 may be a highway or a regular road. If there are no distance markers installed on the road photographed by the in-vehicle camera 3, the nearest distance marker does not need to be identified.

[0055] The embodiments disclosed herein should be considered in all respects as illustrative and not restrictive. The technical features described in each embodiment can be combined with each other, and the scope of the present invention is intended to include all modifications within the claims and scope equivalent to the claims. Furthermore, the independent and dependent claims described in the claims can be combined with each other in any combination, regardless of the form of reference. In addition, the claims use a multi-claim format in which claims refer to two or more other claims (multi-claim format), but are not limited to this. They may also be described using a multi-claim format in which at least one multi-claim refers to another multi-claim (multi-multi-claim format). [Explanation of Symbols]

[0056] 1: Information Processing Device 11: Control Unit 12: Storage section 12a: Recording medium 13: Communications Department 2: Terminal device 21: Terminal Control Unit 22: Storage section 22a: Recording medium 23: Communications Department 24: Display section 25: Input section 3: In-car camera 121:Distance marker DB 122: Compatible DB 123: Driving Image Database M1: Obstructive tree detection model M2: Domain partitioning model M3: Distance calculation model M4: Failure Risk Assessment Module N: Network P: Program Pa: App Program S: Obstructive Tree Detection System

Claims

1. The system acquires an image captured by an in-vehicle camera and location information indicating the location where the image was taken. A learning model that, when given an acquired image as input, outputs the range in the image that contains an obstruction tree that would hinder vehicle movement, is given the image as input and outputs the range that contains the obstruction tree. The image containing the obstructing tree and the positional information of the image containing the obstructing tree are stored in association. A program that instructs a computer to perform a process.

2. The image containing the obstructing tree and the information of the distance marker closest to the location where the image was taken are stored in association. The program according to claim 1.

3. The location of the obstructing tree is displayed on the map by referring to the location information of the image containing the obstructing tree. The program according to claim 1 or 2.

4. Troubled trees are classified into several types, including dead trees or partially dead trees. The learning model determines the type of obstructing tree included in the image. Based on the type of obstructing tree included in the aforementioned image, the method of displaying the location of the obstructing tree on the map will be varied. The program according to claim 3.

5. Based on the reliability of the location information in the image containing the obstructing tree, the method of displaying the location of the obstructing tree on the map will be varied. The program according to claim 3.

6. The system accepts input to correct the result of determining whether or not the aforementioned image contains a problematic tree. Based on whether or not the aforementioned judgment result has been modified, the method of displaying the location of obstructing trees on the map will be changed. The program according to claim 3.

7. The image and the result of determining whether or not the image contains a problematic tree are displayed. The program according to claim 1 or 2.

8. The system accepts input to modify the aforementioned determination result. The program according to claim 7.

9. The learning model is retrained based on training data that associates the aforementioned image with the corrected judgment result. The program according to claim 8.

10. Display the aforementioned image, The system accepts input to correct the range in the displayed image that contains the obstructing tree. The program according to claim 1 or 2.

11. The learning model outputs the tree species of the obstructing tree included in the image. The image above and the species of the obstructing tree included in the image are displayed. The program according to claim 1 or 2.

12. The aforementioned problematic trees are classified into several types, including dead trees, partially dead trees, or overhanging trees. The program according to claim 1 or 2.

13. The system acquires an image captured by an in-vehicle camera and location information indicating the location where the image was taken. A learning model that, when given an acquired image as input, outputs the range in the image that contains an obstruction tree that would hinder vehicle movement, is given the image as input and outputs the range that contains the obstruction tree. The image containing the obstructing tree and the positional information of the image containing the obstructing tree are stored in association. Information processing methods.

14. The system acquires an image captured by an in-vehicle camera and location information indicating the location where the image was taken. A learning model that, when given an acquired image as input, outputs the range in the image that contains an obstruction tree that would hinder vehicle movement, is given the image as input and outputs the range that contains the obstruction tree. The image containing the obstructing tree and the positional information of the image containing the obstructing tree are stored in association. control unit An information processing device equipped with the following features.

Images