Diagnostic system, diagnostic method, and diagnostic program

Abstract

This diagnostic system comprises an acquisition unit, a detection unit, an estimation unit, a state identification unit, a determination unit, and an output unit. The acquisition unit acquires an image capturing a road. The detection unit uses the image to detect equipment that presents information to pedestrians on the road. The estimation unit estimates a rate of coverage of the equipment by objects that appear to overlap with the equipment in the image. The state identification unit identifies states of the apparently overlapping objects. The determination unit determines the necessity to perform work of removing at least part of the objects on the basis of the coverage rate of the equipment estimated by the estimation unit and the states of the objects identified by the state identification unit. The output unit outputs the result of determining the necessity to perform the removal work.

Description

Diagnostic system, diagnostic method, and recording medium

[0001] The present invention relates to a diagnostic system, a diagnostic method, and a recording medium.

[0002] In order for pedestrians to travel safely on roads, equipment on roads that provides information to pedestrians needs to be kept visible to pedestrians. An example of equipment on roads that provides information to pedestrians is signs. Signs may be obscured by trees, making them invisible to pedestrians. Therefore, road administrators, for example, monitor whether signs installed on roads under their management are visible. If the monitoring results show that a sign is obscured by trees, the road administrators may prune the trees to make the sign visible to pedestrians. Furthermore, road administrators may use image recognition technology to determine whether a sign is visible based on images taken by a vehicle traveling on a road under their management.

[0003] In the sign recognition method of Patent Document 1, an image of a sign is synthesized based on two images of a sign captured at different times, and the sign is recognized based on the synthesized image.

[0004] Japanese Patent Application Laid-Open No. 2007-140828

[0005] With the technology described in Patent Document 1, it may be difficult to determine whether or not it is necessary to remove the object covering the sign.

[0006] In order to solve the above-mentioned problems, the present disclosure aims to provide a diagnostic system, etc. that can easily determine whether or not work is needed to remove objects covering equipment that presents information to passersby.

[0007] In order to solve the above problems, the diagnostic system disclosed herein includes an acquisition means for acquiring an image of a road, a detection means for detecting equipment on the road that provides information to passersby based on the image, an estimation means for estimating the coverage rate of the equipment by objects that appear to overlap in the image, a condition identification means for identifying the condition of the objects that appear to overlap, a determination means for determining whether or not work is required to remove at least a portion of the object based on the coverage rate of the equipment estimated by the estimation means and the condition of the object identified by the condition identification means, and an output means for outputting the result of the determination.

[0008] The diagnostic method disclosed herein acquires an image of a road, detects equipment on the road that provides information to passersby based on the image, estimates the coverage rate of the equipment by objects that appear to overlap in the image, identifies the state of the objects that appear to overlap, and determines whether or not work is required to remove at least part of the object based on the estimated coverage rate of the equipment and the state of the identified objects, and outputs the result of the determination.

[0009] The recording medium of the present disclosure non-temporarily records a diagnostic program that causes a computer to execute the following processes: acquiring an image of a road; detecting equipment on the road that provides information to passersby based on the image; estimating the coverage rate of the equipment by objects that appear to overlap in the image; identifying the state of the objects that appear to overlap; determining whether or not work is required to remove at least a portion of the object based on the estimated coverage rate of the equipment and the state of the identified object; and outputting the results of the determination.

[0010] According to the present disclosure, it is possible to easily determine whether or not work is required to remove an object covering equipment that presents information to passersby.

[0011] FIG. 1 is a diagram illustrating an example of a configuration in an embodiment of the present disclosure. FIG. 2 is a diagram illustrating an example of a state in which a sign is covered by tree leaves. FIG. 3 is a diagram illustrating an example of a state in which a sign is covered by tree branches. FIG. 4 is a diagram illustrating a schematic example of an example of capturing an image of a road as a vehicle travels. FIG. 5 is a diagram illustrating an example of a configuration of a diagnostic system in an embodiment of the present disclosure. FIG. 6 is a diagram illustrating an example of a display screen in an embodiment of the present disclosure. FIG. 7 is a diagram illustrating an example of an operation flow of a diagnostic system in an embodiment of the present disclosure. FIG. 8 is a diagram illustrating an example of a hardware configuration of a diagnostic system in an embodiment of the present disclosure.

[0012] An embodiment of the present disclosure will be described in detail with reference to the drawings. FIG. 1 is a diagram illustrating an example of the configuration of a road management system. The road management system includes a diagnostic system 10, an on-board device 20, and a terminal device 30. The diagnostic system 10 is connected to the on-board device 20 via a network, for example. Data input / output between the diagnostic system 10 and the on-board device 20 may be performed via a recording medium. For example, data input / output between the diagnostic system 10 and the on-board device 20 may be performed via a recording medium using a non-volatile semiconductor memory element. The diagnostic system 10 is also connected to the terminal device 30 via the network. There may be multiple on-board devices 20 and multiple terminal devices 30. The number of on-board devices 20 and the number of terminal devices 30 are set as appropriate.

[0013] A road management system is, for example, a system that manages whether equipment that provides information to pedestrians on a road is visible to pedestrians. The diagnostic system 10 estimates the coverage rate of the equipment that provides information to pedestrians by objects based on, for example, photographed images of the road. Then, the diagnostic system 10 determines whether or not work is needed to remove the objects that are covering the equipment that provides information to pedestrians based on, for example, the state of the objects and the coverage rate of the equipment by the objects.

[0014] An example of equipment on a road that provides information to pedestrians is a sign installed on the road. A sign is an object installed to provide pedestrians with information necessary for traveling along the road. A sign, for example, provides pedestrians with information regarding guidance, warnings, restrictions, and instructions. A sign, for example, provides pedestrians with information using text or symbols specified by law. The information provided by a sign is not limited to the above. Equipment on a road that provides information to pedestrians may include traffic lights and convex mirrors. Equipment on a road that provides information to pedestrians may be a display device that displays congestion information, lane restrictions, road closures, road surface conditions, and weather information. Equipment on a road that provides information to pedestrians may also be a sign that guides pedestrians to facilities or destinations. Equipment on a road that provides information to pedestrians is not limited to the above. An example of an object covering the equipment is a tree. An example of an object covering the equipment may be a flag installed on the side of the road. An example of an object covering the equipment is not limited to the above.

[0015] The equipment on the road that presents information to pedestrians is, for example, equipment installed on the shoulder of the road, above the driving lanes, and in the center divider. The equipment on the road that presents information to pedestrians may also include equipment installed outside the road premises. Furthermore, the pedestrian on the road is, for example, the driver of a vehicle traveling on the road. The pedestrian on the road may also be, for example, a passenger other than the driver of a vehicle traveling on the road. The pedestrian on the road may also be a pedestrian. Furthermore, the equipment on the road that presents information to pedestrians may be equipment identified by an imaging device or a sensor for a driving assistance system.

[0016] A state in which an object obscures a facility refers to, for example, a state in which an object exists between a pedestrian and the facility, causing the facility to be obscured by the object from the perspective of the pedestrian. In other words, an object obscuring a facility refers to, for example, a state in which an object exists between a pedestrian and the facility, causing the pedestrian to be unable to see at least a portion of the facility. The state of an object also includes various states of the object. For example, if the object is a tree, the state of the object may be the state of the tree's leaves. For example, if the object is a tree and the facility is a sign, and there are leaves on the branches and the tree has leaves, the pedestrian cannot see the sign through the tree. On the other hand, if there are no leaves on the tree, the pedestrian can see the sign through the tree's branches. A state in which a tree has no leaves also includes, for example, a state in which a tree has only a few leaves on its branches. A state in which a tree has no leaves refers to a state in which the tree is considered to be largely leafless, compared to a state in which it is covered with leaves. When the object is a tree, if it is known whether the tree has leaves or not, it can be determined whether a passerby can see the sign through the tree.

[0017] When estimating tree coverage using image recognition technology, even if there are no leaves and a portion of a sign is visible between the branches, the estimated coverage can be the same as when there are leaves. For example, if the same tree covers a sign, even if the area of ​​the sign is visible between the branches in the leafless winter, the estimated coverage can be the same as when there are leaves in the summer when there are lush leaves. This is because the recognition model used for image recognition identifies, for example, the entire area inside the outer perimeter of the area where branches are present as the tree area. However, when determining whether or not to remove a tree covering a sign, if the tree branches have no leaves, the sign may be visible through the branches. In such a case, if the same criteria are used to determine whether or not to remove the tree, it may be determined that the tree needs to be removed, even though the sign is visible to passersby through the branches and there is no problem even if the tree exists. Therefore, the diagnostic system 10 determines whether or not to remove an object covering equipment that provides information to passersby, for example, based on the condition of the object and the coverage rate of the equipment by the object.

[0018] FIG. 2 is an example of an image showing a sign that is partially obscured by a leafy tree. In the example of the image in FIG. 2, a regulatory sign indicating a speed limit is installed on the left side of the road. Also, in the example of the image in FIG. 2, a tree is present between a pedestrian and the sign, and the sign is obscured by leaves growing on the tree branches. In the example of the image in FIG. 2, the number indicated by the sign is not visible to the pedestrian due to the leaves of the tree. In the example of the screen in FIG. 2, for example, the road administrator needs to prune the trees to remove the portion obscuring the sign, thereby making the number indicated by the sign visible to the pedestrian.

[0019] FIG. 3 is an example of an image showing a sign that is partially obscured by the branches of a leafless tree. In the example image of FIG. 3, a regulatory sign indicating a speed limit is installed on the left side of the road. Also, in the example image of FIG. 3, a tree is present between a pedestrian and the sign, and the sign is obscured by the tree branch. In the example screen state of FIG. 3, a pedestrian can see the number indicated by the sign through the branches. Therefore, in the example screen state of FIG. 3, the road administrator does not need to immediately prune the tree.

[0020] However, the recognition model used for image recognition may determine that the area between the branches is also covered by trees in the example image of FIG. 3 . If it is determined that the area between the branches is also covered by trees, the coverage rate of the sign by trees in the example screen of FIG. 2 may be approximately the same as the coverage rate of the sign by trees in the example screen of FIG. 3 . Therefore, if a single threshold value is set for the coverage rate to determine whether or not a tree needs to be removed, a low threshold may result in a determination that the tree needs to be removed even though the sign is visible between the branches and pruning is unnecessary, as shown in FIG. 3 . On the other hand, a high threshold may result in a determination that the tree needs to be removed even though the sign is obscured by leaves growing on the tree branches and cannot be seen, as shown in FIG. 2 . Therefore, it is preferable to set a lower threshold value for determining whether or not a tree needs to be removed when there are leaves on the tree branches than when there are only branches and no leaves.

[0021] The threshold for determining whether tree removal work is necessary is set as a criterion for determining whether tree removal work is necessary, for example, using the coverage rate of signs by trees. The road management system determines that tree removal work is necessary, for example, when the coverage rate of signs by trees is equal to or greater than the threshold. Furthermore, in the road management system, for example, the threshold for determining whether tree removal work is necessary when the tree has no leaves is set higher than when the tree has leaves. In this way, the road management system can appropriately determine whether tree removal work is necessary by determining whether tree removal work is necessary based on a threshold according to the condition of the leaves on the tree, for example.

[0022] The diagnostic system 10 identifies the state of objects covering facilities based on, for example, photographed images of a road. The diagnostic system 10 also estimates the coverage rate of facilities covered by objects based on the photographed images of the road. The diagnostic system 10 then determines whether work to remove the objects covering the facilities is necessary based on the estimated coverage rate and the state of the objects. The on-board device 20 is, for example, a device that captures images using an imaging device, which the diagnostic system 10 uses to estimate the coverage rate of facilities. The terminal device 30 is, for example, a terminal device used by a road administrator to operate the diagnostic system 10 and confirm whether or not object removal is necessary. The road administrator is, for example, an entity that manages roads to ensure safe road traffic. The road administrator, for example, monitors and maintains road-related facilities. The road administrator, for example, refers to the results of the determination made by the diagnostic system 10 and determines whether or not work to remove the objects covering the facilities is necessary.

[0023] FIG. 4 is a diagram schematically illustrating an example of monitoring the state of road signs based on images captured by a camera mounted on a vehicle. In the example of FIG. 4, a vehicle is traveling on a road. Also, in the example of FIG. 4, signs indicating speed limits are installed on the road. In the example of FIG. 4, an in-vehicle device 20 is installed on the vehicle. The in-vehicle device 20, for example, uses the camera to capture an image of the road that shows the signs installed on the road. The in-vehicle device 20 outputs the captured image to, for example, the diagnostic system 10. For example, a drive recorder is used as the in-vehicle device 20. The in-vehicle device 20 may be something other than a drive recorder.

[0024] The diagnostic system 10 identifies the condition of trees based on images acquired from the in-vehicle device 20. The diagnostic system 10 also estimates the coverage rate of trees on signs. The diagnostic system 10 then determines whether or not tree removal is necessary based on the condition of the trees and the coverage rate of trees on signs. The diagnostic system 10 outputs the determination result of whether or not tree removal is necessary to, for example, the terminal device 30. The terminal device 30 outputs the determination result of whether or not tree removal is necessary to, for example, a display device. The road administrator, for example, refers to the determination result of whether or not tree pruning is necessary and determines whether or not tree removal is necessary by pruning.

[0025] Here, the configuration of the diagnostic system 10 will be described. Fig. 5 is a diagram showing an example of the configuration of the diagnostic system 10. The diagnostic system 10 basically includes an acquisition unit 11, a detection unit 12, a state identification unit 13, an estimation unit 14, a determination unit 15, and an output unit 16. The diagnostic system 10 also includes, for example, a storage unit 17.

[0026] The acquisition unit 11 acquires a photographed image of a road. The acquisition unit 11 acquires the photographed image of a road from, for example, the in-vehicle device 20. The photographed image of a road is an image that shows equipment on the road that provides information to pedestrians. The photographed image of a road is, for example, captured by a camera mounted on a vehicle. The camera mounted on a vehicle captures the road, for example, so that equipment on the road that provides information to pedestrians is captured in the image. The photographed image of a road is, for example, a video captured by a drive recorder. The acquisition unit 11 may acquire the photographed image of a road from the in-vehicle device 20 via a recording medium. For example, a recording medium including a non-volatile semiconductor memory element is used as the recording medium. The recording medium is not limited to the above. The acquisition unit 11 may also acquire the photographed image of a road from a server connected to a network. The acquisition unit 11 may also acquire information on the position of the vehicle from the in-vehicle device 20.

[0027] The detection unit 12 detects equipment on a road that provides information to pedestrians based on a captured image of the road. The detection unit 12 detects, for example, equipment on a road that appears in a captured image of the road that provides information to pedestrians. The detection unit 12 detects, for example, equipment on a road that appears in the image that provides information to pedestrians by using a recognition model.

[0028] The recognition model is, for example, a learning model that identifies equipment that provides information to passersby on a road that is shown in an image. The recognition model may also be a learning model that further identifies the state of an object covering the equipment. The recognition model is, for example, a machine learning model using a neural network. The recognition model is generated by learning the relationship between an image that shows equipment that provides information to passersby on a road and the type of equipment. The recognition model may also be one that further learns the relationship between an image that shows an object that is on or near the road and the state of the object. The learning data used to generate the recognition model is not limited to the above. The recognition model is, for example, generated in a system external to the diagnostic system 10.

[0029] When the image of a road is a video of a continuous shot of the road, the detection unit 12 selects a frame image suitable for checking the state of coverage of the equipment by an object from among multiple frames showing the same equipment. An image suitable for checking the state of coverage of the equipment by an object is, for example, an image in which the size of the equipment shown in the image is within a standard range. The size of the equipment shown in the image is indicated, for example, by the number of pixels in the image. The size of the equipment shown in the image is set, for example, for each piece of equipment, based on the number of pixels in the image and the lens magnification of the shooting device. When there are multiple images in which the size of the equipment shown in the image is within a standard range, the detection unit 12 selects the image with the largest size of the equipment among the images within the standard range. Furthermore, an image suitable for checking the state of coverage of the equipment by an object may be an image in which the distance between the shooting point and the equipment is within a standard range. The distance between the shooting point and the equipment is calculated, for example, based on the vehicle speed and the time difference between the shooting time of each frame and the shooting time of the frame taken closest to the equipment. The frame taken closest to the equipment is, for example, a frame in which no equipment is shown after that frame in the chronological order. The criteria for selecting an image frame suitable for checking the state of coverage of the equipment by an object are not limited to the above. The process of selecting an image frame suitable for checking the state of coverage of the equipment by an object may be performed by, for example, the determination unit 15.

[0030] When multiple pieces of equipment are simultaneously captured in a single frame of image, the detection unit 12 distinguishes and detects each piece of equipment based on, for example, at least one of the position, shape, and design of the equipment in the image. Furthermore, when equipment is captured in each frame of a time-series image, the detection unit 12 may detect that each piece of equipment is a different piece of equipment if the equipment disappears from the image for a set number of consecutive frames and then the same type of equipment appears in the image. For example, when multiple frames in which equipment is captured are followed by one frame in which the equipment is not captured, and then multiple frames in which the same type of equipment is captured, the detection unit 12 detects that the equipment captured in the previous and following frames is the same equipment. The set value for the number of frames in which equipment is not captured when equipment captured in the previous and following frames is considered to be the same equipment is set based on, for example, the vehicle speed and the frame rate of the images capturing the road. Furthermore, when multiple pieces of equipment are captured in the time-series image, the detection unit 12 may identify each piece of equipment based on vehicle position information acquired by the acquisition unit 11. The information on the position of each sign may be set by, for example, a road administrator. The detection unit 12 may identify each facility based on the information on the vehicle position acquired by the acquisition unit 11, the vehicle speed, and the vehicle acceleration.

[0031] The state identification unit 13 identifies the state of an object that appears to overlap the facility detected by the detection unit 12 in an image of a road. The state identification unit 13, for example, identifies whether the state of the object is a state in which the facility is visible through the object. For example, if the object is a tree and the facility is a sign, the state of the object is the presence or absence of leaves on the tree. Furthermore, if the object is a tree, the state of the object may be the thickness of the tree's branches. If the object is a tree, the state of the object may be whether the tree is dead or not. The state identification unit 13, for example, identifies the state of the object covering the facility based on the image identification result by the detection unit 12. Furthermore, the state identification unit 13 may identify the state of the object using a recognition model that differs from the recognition model used by the detection unit 12 in at least one of training data and algorithm.

[0032] The state identification unit 13 identifies the state of an object based on, for example, the color features of the overlapping portion of the object in the identification result of the recognition model. For example, the state of a tree is identified based on whether the chromaticity of the portion of the tree where the sign overlaps in the object image is the chromaticity of the leaf or the chromaticity of the branch. The range of chromaticity of the leaf and the range of chromaticity of the branch are set based on, for example, the vegetation around the road. The state identification unit 13 identifies the tree as having no leaves on its branches when the chromaticity identified from the image is within the range of the chromaticity of the branch. Furthermore, the state identification unit 13 identifies the tree as having leaves on its branches when the chromaticity of the portion of the tree where the sign overlaps in the image is within the range of the chromaticity of the leaf.

[0033] The state identification unit 13 may also identify the state of leaves growing on branches based on color characteristics of a portion of the image where the identified tree overlaps with a sign. The state identification unit 13, for example, identifies the state of leaves growing on branches in a plurality of stages based on the chromaticity of the portion of the image where the identified tree overlaps with a sign. The plurality of stages for identifying the state of leaves is set, for example, by dividing the relationship between the chromaticity of the leaves and the color of the branch into a plurality of intervals based on the chromaticity in the XYZ color system. The state identification unit 13, for example, identifies the amount of leaves growing on branches as one of a plurality of stages based on which of a plurality of intervals the chromaticity of the portion of the image where the identified tree overlaps with a sign falls into.

[0034] The state identification unit 13 may identify the thickness of tree branches based on changes in at least one of brightness and color in the image. The branch thickness is expressed, for example, using the number of pixels in the image. The state identification unit 13 extracts the amount of change in at least one of brightness and chromaticity in one direction in the image. Then, the state identification unit 13 identifies the thickness of tree branches based on the cycle at which changes in at least one of brightness and chromaticity occur in the area where the tree and the sign overlap.

[0035] The state identification unit 13 may identify the state of an object by identifying whether the sky is visible through the object. For example, if the object is a tree and the tree is a sign, and the sky is visible through the object, the sign can be seen through the object. The state of the object may be whether the object is temporarily present or always present. For example, the state of the object may be whether it is temporarily installed for work or permanently installed. The state of the object may also be a state in which the degree of covering of equipment changes due to wind or the object's own weight. For example, if the object is a flag, the state may change between a state in which the flag covers the equipment behind it as the flag moves in the wind, making the equipment invisible, and a state in which the equipment is not covered and can be seen. The state of the object may also be whether it remains in the same state throughout the year or a state in which it is only in a specific season. For example, if the object is a tree that grows in the summer, the state identification unit 13 identifies it as a state in which it grows only during the summer. For example, if the object is snow, the state identifying unit 13 identifies it as existing only during the winter period or for a few days.

[0036] The estimation unit 14 estimates the coverage rate of the facility by the object based on an image of the road. The coverage rate of the facility is, for example, the ratio of the size of the portion covered by the object to the size of the display surface of the facility in the image. The size of the portion covered by the object includes, for example, the portion of the facility visible through the gaps between the objects. For example, if the object is a tree and the facility is a sign, the size of the portion covered by the object also includes the sign visible between the tree branches. The estimation unit 14 calculates the size of the facility when it is not covered based on, for example, the dimensions of the display surface of the facility. The size of the display surface of the facility is represented, for example, by the number of pixels in the image. For example, the estimation unit 14 calculates the dimensions of the display surface of the facility based on the shape of the display surface of the facility detected by the detection unit 12, and calculates the size of the display surface of the facility. For example, if the facility is circular and the detection unit 12 detects a portion of the periphery of the circle, the estimation unit 14 considers the identified portion to be a circle including the periphery and calculates the diameter of the circle. The estimation unit 14 then calculates the size of the circle based on the diameter of the circle. The estimation unit 14 calculates the coverage rate of the equipment by the object based on the calculated size of the display surface of the equipment and the size of the display surface of the equipment detected by the detection unit 12. The estimation unit 14 calculates the coverage rate of the equipment by the object, for example, by dividing the size of the display surface of the equipment detected by the detection unit 12 by the size of the display surface of the equipment when it is not covered by the object. Without being limited to the above, the coverage rate may be expressed as, for example, two values, "high" or "low," or may be expressed in multiple levels.

[0037] The determination unit 15 determines whether or not work to remove at least a portion of the object is necessary based on the coverage rate of the equipment estimated by the estimation unit 14 and the state of the object identified by the state identification unit 13. For example, when the coverage rate of the equipment estimated by the estimation unit 14 is equal to or greater than a threshold corresponding to the state of the object identified by the state identification unit 13, the determination unit 15 determines that work to remove at least a portion of the object is necessary. For example, the state identification unit 13 sets a threshold for the coverage rate of the equipment used to determine whether or not work is necessary based on the identification result of the state of the object. Then, when the coverage rate of the equipment estimated by the estimation unit 14 is equal to or greater than the set threshold, the state identification unit 13 determines that work to remove at least a portion of the object is necessary. For example, when the object is a tree, the work to remove at least a portion of the object is an operation of selecting branches covering the equipment. The work to remove at least a portion of the object may be an operation to remove the entire object. For example, when the object is a tree, the work to remove at least a portion of the object may be an operation of cutting or transplanting the trunk of the tree.

[0038] For example, the determination unit 15 sets a threshold value of the coverage rate of the equipment used to determine whether or not work is necessary when the state of the object is such that the equipment can be seen through the object, higher than when the equipment cannot be seen through the object. The entity that sees the object is, for example, a passerby on the road. The threshold value used to determine whether work to remove at least a part of the object is necessary is defined in a table that shows the relationship between the state of the object and the threshold value.

[0039] For example, when the object is a tree and the state of the object is the presence or absence of leaves, the determination unit 15 sets the threshold value of the sign coverage used to determine whether or not work is required to be performed such that the threshold value when leaves are present is 40 percent and the threshold value when no leaves are present is 60 percent. In this case, the unit of coverage is percent.

[0040] The determination unit 15 may set the threshold of the equipment coverage rate used to determine whether work is required in multiple stages depending on the state of the object. For example, if the object is a tree, the determination unit 15 may set the threshold in multiple stages depending on the amount of leaves on the tree. For example, the determination unit 15 may set different thresholds for a state where the tree branches have no leaves, a state where leaves are growing on some of the tree branches, and a state where the entire tree branches are covered with leaves. The determination unit 15 may also set the threshold of the equipment coverage rate used to determine whether work is required using a function. For example, if the object is a tree, the determination unit 15 sets the threshold of the equipment coverage rate using a function in which the amount of tree leaves is an explanatory variable and the threshold is a target variable. The amount of tree leaves is estimated by the state identification unit 13 based on the chromaticity of the tree shown in the image, for example.

[0041] The determination unit 15 may set a threshold value for the coverage rate of equipment used to determine whether work is required, depending on the importance of the equipment. The determination unit 15 sets the threshold value for the coverage rate of equipment, for example, using a table indicating the relationship between the type of equipment and the state of the object, and the threshold value. For example, when the equipment is a sign, the determination unit 15 sets different threshold values ​​for guide signs and regulatory signs. For example, the determination unit 15 sets a lower threshold value for regulatory signs such as "Stop" than for guide signs, because not being able to see the sign can lead to an accident.

[0042] The determination unit 15 may set the threshold value of the coverage rate of the equipment used to determine whether or not work is required higher when the sky is visible behind the object than when the sky is not visible. This is because when the sky is visible behind the object, it is more likely that the facility covered by the object can be seen. Furthermore, the determination unit 15 may set the threshold value higher when the object is a temporary object than when it is a permanent object.

[0043] The determination unit 15 may further determine the priority of the work to remove at least a portion of the object. The determination unit 15 determines the priority of the work based on, for example, the ratio of the coverage rate of the equipment by the object to a threshold value of the coverage rate used to determine whether the work is necessary. The determination unit 15 calculates the priority of whether the work is necessary by, for example, normalizing the coverage rate of the equipment by the object based on the threshold value of the coverage rate used to determine whether the work is necessary. The determination unit 15 calculates the priority of whether the work is necessary by, for example, normalizing the value of the coverage rate of the equipment by the object by the threshold value of the coverage rate used to determine whether the work is necessary. Then, the determination unit 15 determines that the priority of the work for equipment with a high calculated priority value is high. The priority of the work to remove at least a portion of the object may be calculated by further multiplying the value of the coverage rate of the equipment by the object normalized using the threshold value used to determine whether the work is necessary by a value indicating the importance of the equipment.

[0044] Furthermore, the determination unit 15 may determine whether or not work to remove at least a portion of the object is necessary based on an index obtained by normalizing the coverage rate of the equipment by the object using a value set based on the state of the object. For example, the determination unit 15 normalizes the value of the coverage rate of the equipment by the object by a value set based on the state of the object. The value set based on the state of the object is set higher, for example, when the equipment is visible through the object than when the equipment is not visible through the object. Therefore, when the coverage rate values ​​estimated by the estimation unit 14 are the same, the normalized index will be lower when the equipment is visible through the object than when the equipment is not visible through the object.

[0045] The output unit 16 outputs the result of the determination made by the determination unit 15. The output unit 16 outputs, for example, the result of the determination as to whether or not work to remove at least a portion of the object is necessary. The output unit 16 outputs the result of the determination made by the determination unit 15 to, for example, the terminal device 30. The output unit 16 may output the result of the determination made by the determination unit 15 to a display device (not shown) connected to the diagnostic system 10.

[0046] The output unit 16 outputs, for example, for each piece of equipment, the result of the determination as to whether or not work is required to remove at least a portion of the object. When a sign has been installed, the output unit 16 may output the identifier or location information of the equipment and the result of the determination as to whether or not work is required to remove at least a portion of the object. Furthermore, the output unit 16 may output the result of the determination as to whether or not work is required to remove at least a portion of the object by superimposing it on a position on the map corresponding to the installation location of the equipment. In addition to the result of the determination, the output unit 16 may output an image used in the determination as to whether or not work is required to remove at least a portion of the object. In addition to the result of the determination, the output unit 16 may output an image used in the determination as to whether or not work is required to remove at least a portion of the object and a coverage rate of the equipment by the object. In addition to the result of the determination, the output unit 16 may output the coverage rate of the equipment by the object. For example, the output unit 16 may output whether or not work is required to remove at least a portion of the object, an image showing the equipment, and a coverage rate of the equipment by the object. Furthermore, the determination unit 15 may further output the state of the object for each piece of equipment. The determination unit 15 may further output a threshold value used to determine whether or not an operation to remove at least a part of the object is necessary.

[0047] If the object is a tree, the determination unit 15 may further use at least one of the planting date and pruning history of trees around the equipment to determine whether or not work to remove at least a portion of the tree is necessary. The planting date of a tree is, for example, information indicating when the tree was planted. The planting date of a tree may also be the age of the tree. The pruning history is, for example, a record of past branch cutting work. The determination unit 15 determines whether or not work to remove at least a portion of a tree is necessary for equipment around the equipment where the number of days since the planting of trees around the equipment or the last pruning is equal to or greater than a standard, using a lower threshold than for equipment where the number of days since the last pruning is less than the standard. Furthermore, if tree pruning work is scheduled around the equipment, the determination unit 15 may exclude the equipment from targets for determining whether or not work to remove at least a portion of the tree is necessary.

[0048] The output unit 16 may output a result of the determination as to whether or not work to remove at least a portion of the object is necessary for the equipment installed at each of the multiple locations. The output unit 16 may output an image used in the determination as to whether or not work to remove at least a portion of the object is necessary for the equipment installed at each of the multiple locations, in addition to the determination result. The output unit 16 may output a coverage rate of the equipment by the object for the equipment installed at each of the multiple locations, in addition to the determination result. The output unit 16 may output an image used in the determination as to whether or not work to remove at least a portion of the object is necessary for the equipment installed at each of the multiple locations, in addition to the determination result.

[0049] When the priority of the work to remove at least a portion of the object is determined, the output unit 16 may further output the priority of the work. Also, the output unit 16 may output at least one of an image and a coverage rate of equipment that requires the work to remove at least a portion of the object in descending order of priority.

[0050] 6 and 7 show examples of display screens that display the results of a determination on whether or not work is needed to remove trees obscuring signs. The example display screen of FIG. 6 displays, as a determination result, locations where work is needed to remove trees obscuring signs. In the example display screen of FIG. 6, an image of a location where tree removal is needed is displayed as "Location where pruning work is needed." In the example display screen of FIG. 6, an image of a sign installed at point S on road A and an image of a sign installed at point B are displayed. In the example display screen of FIG. 6, each of the signs shown in the image is obscured by a tree. In the example display screen of FIG. 6, the sign at point S is obscured by tree leaves. In addition, in the example display screen of FIG. 6, the sign at point B is obscured by leafless tree branches. In the example display screen of FIG. 6, the numbers displayed on the signs at points S and B are not visible to passersby. In addition, in the example display screen of FIG. 6, the sign coverage rate at point S is displayed as 38. In the example display screen of FIG. 6, the sign coverage rate at point B is displayed as 79. By referring to a display screen such as the example of FIG. 6, a road administrator can determine, for example, signs indicating that trees need to be pruned.

[0051] The example of the display screen in Fig. 7 is a display screen that displays, as a determination result, points where work to remove trees covering signs is required and points where work is not required. In the example of the display screen in Fig. 7, an image of a point where tree removal is required is displayed as "point where pruning work is required." In addition, in the example of the display screen in Fig. 7, an image of a point where tree removal is required is displayed as "point where pruning work is not required." In the example of the display screen in Fig. 7, an image of a sign installed at point S on road A and an image of a sign installed at point B are displayed as images of points where tree removal work is required. In the example of the display screen in Fig. 7, an image of a sign at point C on road A is displayed as an image of a point where tree removal work is not required.

[0052] In the example of the display screen in Figure 7, each of the signs shown in the image is obscured by trees. In the example of the display screen in Figure 7, the sign at point S is obscured by tree leaves. Also, in the example of the display screen in Figure 7, the signs at points B and C are obscured by leafless tree branches. In the example of the display screen in Figure 7, the signs at points S and B are in a state where passersby cannot see the numbers displayed on the signs. On the other hand, in the example of the display screen in Figure 7, the sign at point C is in a state where passersby can see the numbers displayed on the sign through the branches.

[0053] In the example display screen of FIG. 7 , the sign coverage rate at point S is displayed as 38. Also, in the example display screen of FIG. 7 , the sign coverage rate at point B is displayed as 79. Also, in the example display screen of FIG. 7 , the sign coverage rate at point C is displayed as 55. In the example display screen of FIG. 7 , the sign coverage rate at point C is higher than the sign coverage rate at point S. However, because the tree covering the sign at point C is leafless, the threshold for determining whether work is necessary is set higher than that for the sign at point S. Therefore, the sign at point C is determined not to require tree removal work. In this way, by determining whether tree removal work is necessary using a threshold set according to the condition of the tree leaves, the diagnostic system 10 can appropriately determine whether tree removal work is necessary according to the condition of the tree. Furthermore, by referring to the display screen of the example of FIG. 7 , a road administrator can, for example, determine which signs require tree pruning while confirming the validity of the determination made by the diagnostic system 10.

[0054] The memory unit 17 stores, for example, data related to a process for determining whether or not an operation to remove at least a portion of an object is necessary. The memory unit 17 stores, for example, an image of a road. The memory unit 17 stores, for example, a recognition model. The recognition model may be stored in a storage means other than the memory unit 17. The memory unit 17 may also store the result of selecting an image to be used for checking the state of coverage of a sign by an object. The memory unit 17 stores, for example, an image used for determining whether or not an operation to remove at least a portion of an object is necessary. The memory unit 17 stores, for example, the sign coverage rate estimated by the estimation unit 14.

[0055] The in-vehicle device 20 includes, for example, a camera that captures images of the road. The camera of the in-vehicle device 20 captures, for example, images of the road including equipment that provides information to pedestrians. The camera continuously captures, for example, images of the road while the vehicle is traveling. The in-vehicle device 20 includes, for example, a camera that captures images of the front of the vehicle. The front of the vehicle is, for example, the direction in front of the driver when sitting in the driver's seat. The camera of the in-vehicle device 20 may also capture images of the rear of the vehicle. The rear of the vehicle is, for example, the direction behind the driver when sitting in the driver's seat. The in-vehicle device 20 outputs the captured images of the road and data related to the vehicle's speed to the diagnostic system 10, for example, via a network. In addition, when data is input / output to / from the diagnostic system 10 via a recording medium, the in-vehicle device 20 includes, for example, a slot for inserting a removable recording medium.

[0056] The in-vehicle device 20 may add information about the location where the image was taken to the captured image. The in-vehicle device 20 may identify the location of the vehicle when the image was taken, for example, using GNSS (Global Navigation Satellite System). The in-vehicle device 20 may identify the location of the vehicle based on a beacon containing location information. The in-vehicle device 20 may identify the location where the image was taken based on map information and the travel distance from the location where the vehicle's location was identified. Furthermore, the in-vehicle device 20 outputs the captured image to, for example, the diagnosis system 10. For example, a drive recorder is used as the in-vehicle device 20. The in-vehicle device 20 is not limited to a drive recorder.

[0057] The terminal device 30, for example, acquires the result of the determination as to whether or not work to remove at least a portion of the object is necessary. The terminal device 30 acquires the result of the determination as to whether or not work to remove at least a portion of the object is necessary, for example, from the diagnostic system 10. Then, the terminal device 30 outputs the result of the determination as to whether or not work to remove at least a portion of the object is necessary, for example, to a display device (not shown). The terminal device 30 acquires, for example, an image on which the determination as to whether or not work to remove at least a portion of the object is necessary and the coverage rate of the marker by the object in the image. Then, the terminal device 30 outputs the result of the determination as to whether or not work to remove at least a portion of the object is necessary, the image used in the determination, and the coverage rate of the equipment by the object, for example, to a display device (not shown).

[0058] For example, a personal computer, a tablet computer, or a smartphone can be used as the terminal device 30. The terminal device 30 is not limited to the above examples.

[0059] The following describes the operation of the diagnostic system 10 to determine whether or not it is necessary to remove an object covering a sign. Fig. 8 shows an example of the operation flow when the diagnostic system 10 determines whether or not it is necessary to remove an object covering a sign.

[0060] The acquisition unit 11 acquires an image of a road from the in-vehicle device 20, for example (step S11).

[0061] When the image of the road is acquired, the detection unit 12 detects facilities on the road that provide information to pedestrians based on the image of the road (step S12).

[0062] If the detection result of the facility that presents information to passersby shows that an object appears to overlap the facility (Yes in step S13), the estimation unit 14 estimates the coverage rate of the facility by the object that appears to overlap in the image (step S14).

[0063] Once the coverage rate of the equipment is estimated, the state identification unit 13 identifies the state of objects that appear to overlap the detected equipment in the image (step S15).

[0064] Once the state of the object has been identified, the judgment unit 15 judges whether or not work to remove at least a portion of the object is necessary based on the state of the object identified by the state identification unit 13, the coverage rate of the equipment estimated by the estimation unit 14, and the state of the object identified by the state identification unit 13 (step S16).

[0065] When it is determined whether or not the work of removing at least a part of the object is necessary, the output unit 16 outputs the result of the determination of whether or not the work is necessary (step S17). The output unit 16 outputs the result of the determination of whether or not the work is necessary to, for example, the terminal device 30.

[0066] Also, in step S13, if the equipment is not covered by an object (No in step S13), the output unit 16 outputs, as a result of the judgment, information indicating that, for example, work to remove at least a part of the object is not necessary (step S17).

[0067] The diagnostic system 10 identifies the state of an object covering a facility that provides information to pedestrians on a road in a captured image. The diagnostic system 10 also estimates the coverage rate of the facility by the object. The diagnostic system 10 then determines whether or not work to remove at least a portion of the object is necessary based on the state of the object and the estimated coverage rate of the facility.

[0068] For example, if the facility providing information to pedestrians is a road sign and the object is a tree, the diagnostic system 10 identifies the condition of the tree. The condition of the tree, for example, is determined by whether the tree has leaves on its branches. The diagnostic system 10 also estimates the tree's coverage of the sign. The diagnostic system 10 then determines whether or not to remove at least a portion of the tree based on a threshold value set according to the tree's condition and the tree's coverage of the sign. For example, if the tree has no leaves, the threshold value is set lower than if the tree has leaves. This is because, even if the tree has no leaves and the coverage is estimated to be the same as if the tree has leaves, the sign may be visible through the branches. For example, if the tree's coverage of the sign is equal to or greater than the threshold value, the diagnostic system 10 determines that at least a portion of the tree needs to be removed. For example, a road administrator may refer to the determination result of the diagnostic system 10 to determine whether or not to prune the tree. In this way, by using the diagnostic system 10, it is possible to easily determine whether or not to remove an object covering the facility providing information to pedestrians.

[0069] In addition, when an image indicating whether or not work is required is output along with the judgment result, the manager of the road equipment can, for example, refer to an image showing equipment covered by an object, and determine whether or not work is required while confirming the validity of the judgment result made by the diagnostic system 10.

[0070] Furthermore, when calculating the priority of work to remove at least a portion of an object, the manager of road facilities can easily determine the order of work by, for example, referring to the priority.

[0071] The processes in the diagnostic system 10 may be distributed and executed among a plurality of information processing devices connected via a network. For example, the processes in the detection unit 12, the state identification unit 13, and the estimation unit 14 and the process in the determination unit 15 may be executed in different information processing devices. It may be appropriately set which of the plurality of information processing devices executes each process in the diagnostic system 10.

[0072] Each process in the diagnostic system 10 can be realized by executing a computer program on a computer. Fig. 9 shows an example of the configuration of a computer 100 that executes a computer program that performs each process in the diagnostic system 10. The computer 100 includes a CPU (Central Processing Unit) 101, a memory 102, a storage device 103, an input / output I / F (Interface) 104, and a communication I / F 105.

[0073] The CPU 101 reads and executes computer programs for performing each process from the storage device 103. The CPU 101 may be configured with a combination of multiple CPUs. Furthermore, the CPU 101 may be configured with a combination of a CPU and another type of processor. For example, the CPU 101 may be configured with a combination of a CPU and a graphics processing unit (GPU). The memory 102 is configured with a dynamic random access memory (DRAM) or the like, and temporarily stores computer programs executed by the CPU 101 and data being processed. The storage device 103 stores the computer programs executed by the CPU 101. The storage device 103 is configured with, for example, a non-volatile semiconductor memory element. Other storage devices such as a hard disk drive may also be used for the storage device 103. The input / output I / F 104 is an interface that receives input from an operator and outputs display data, etc. The communication I / F 105 is an interface that transmits and receives data between the in-vehicle device 20, the terminal device 30, and other information processing devices. Furthermore, the terminal device 30 may have a configuration similar to that of the computer 100.

[0074] The computer program used to execute each process can also be stored and distributed on a computer-readable recording medium that non-temporarily stores data. Examples of recording media that can be used include magnetic tapes for recording data and magnetic disks such as hard disks. Optical disks such as CD-ROMs (Compact Disc Read Only Memory) can also be used as recording media. Non-volatile semiconductor storage devices can also be used as recording media.

[0075] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0076] [Supplementary Note 1] A diagnostic system comprising: an acquisition means for acquiring an image of a road; a detection means for detecting equipment on the road that provides information to passersby based on the image; an estimation means for estimating a coverage rate of the equipment by objects that appear to overlap in the image; a state identification means for identifying the state of the objects that appear to overlap; a determination means for determining whether or not work is required to remove at least a portion of the object based on the coverage rate of the equipment estimated by the estimation means and the state of the object identified by the state identification means; and an output means for outputting the result of the determination.

[0077] [Supplementary Note 2] The diagnostic system according to Supplementary Note 1, wherein the determination means determines that work to remove at least a portion of the object is necessary when the coverage rate of the equipment is equal to or greater than a threshold value set according to a state of the object.

[0078] [Supplementary Note 3] The diagnostic system described in Supplementary Note 2, wherein the object is a tree, and the condition of the object is the condition of the leaves of the tree, and the determination means determines that work to remove at least a portion of the tree is necessary when the coverage rate of the equipment by the tree is equal to or greater than a threshold set according to the condition of the leaves of the tree.

[0079] [Supplementary Note 4] The diagnostic system according to Supplementary Note 3, wherein the threshold value when the tree has no leaves is set higher than the threshold value when the tree has leaves.

[0080] [Supplementary Note 5] The diagnostic system according to Supplementary Note 3 or 4, wherein the threshold value when the state identification means identifies the sky as the background of the trees is set higher than the threshold value when the sky is not identified.

[0081] [Supplementary Note 6] The diagnostic system according to any one of Supplementary Notes 3 to 5, wherein the determining means further uses at least one of the planting date and pruning history of the tree to determine whether or not work to remove at least a portion of the tree is necessary.

[0082] [Supplementary Note 7] The diagnostic system described in Supplementary Note 2, wherein the object is a tree, and the condition of the object is the thickness of the tree's branches, and the determination means determines that work to remove at least a portion of the tree is necessary when the coverage rate of the equipment by the tree is equal to or greater than a threshold set according to the thickness of the tree's branches.

[0083] [Supplementary Note 8] The diagnostic system according to Supplementary Note 7, wherein the threshold value when the thickness of the tree branch is less than the standard is set higher than the threshold value when the thickness of the tree branch is equal to or greater than the standard.

[0084] [Supplementary Note 9] The diagnostic system according to any one of Supplementary Notes 1 to 8, wherein the determining means further determines a priority of an operation to remove at least a portion of the object.

[0085] [Supplementary Note 10] The diagnostic system according to Supplementary Note 1, wherein the determining means determines the priority of the work based on a ratio of a coverage rate of the equipment by the object to a threshold value of the coverage rate used to determine whether or not the work is necessary.

[0086] [Supplementary Note 11] A diagnostic method comprising: acquiring an image of a road; detecting equipment on the road that provides information to pedestrians based on the image; estimating a coverage rate of the equipment by objects that appear to overlap in the image; identifying the state of the objects that appear to overlap; determining whether or not work to remove at least a part of the objects is necessary based on the determined coverage rate of the equipment and the identified state of the objects; and outputting the result of the determination.

[0087] [Supplementary Note 12] A recording medium that non-temporarily records a diagnostic program that causes a computer to execute the following steps: acquiring an image of a road; detecting equipment on the road that provides information to pedestrians based on the image; estimating a coverage rate of the equipment by objects that appear to overlap in the image; identifying the state of the objects that appear to overlap; determining whether or not work to remove at least a part of the objects is necessary based on the estimated coverage rate of the equipment and the identified state of the objects; and outputting the result of the determination.

[0088] The present disclosure has been described above using the above-described embodiments as examples. However, the present disclosure is not limited to the above-described embodiments. That is, the present disclosure can be applied in various aspects that can be understood by a person skilled in the art within the scope of the present disclosure.

[0089] REFERENCE SIGNS LIST 10 Diagnostic system 11 Acquisition unit 12 Detection unit 13 State identification unit 14 Estimation unit 15 Determination unit 16 Output unit 17 Storage unit 20 In-vehicle device 30 Terminal device 100 Computer 101 CPU 102 Memory 103 Storage device 104 Input / output I / F 105 Communication I / F

Claims

1. an acquisition means for acquiring an image of a road; a detection means for detecting a facility for presenting information to pedestrians on a road based on the image; an estimation means for estimating a coverage rate of the facility by objects that appear to overlap in the image; a state identifying means for identifying a state of the objects that appear to overlap; a determination means for determining whether or not an operation to remove at least a portion of the object is necessary based on the coverage rate of the equipment estimated by the estimation means and the state of the object identified by the state identification means; an output means for outputting the result of the determination; A diagnostic system comprising:

2. The determination means determines that an operation to remove at least a part of the object is necessary when the coverage rate of the equipment is equal to or greater than a threshold value set according to the state of the object. The diagnostic system of claim 1 .

3. the object is a tree, and the state of the object is the state of leaves of the tree; the determination means determines that work to remove at least a portion of the trees is necessary when the coverage rate of the facilities by the trees is equal to or greater than a threshold set according to the state of leaves of the trees; The diagnostic system of claim 2 .

4. the threshold value when the tree has no leaves is set higher than the threshold value when the tree has leaves, The diagnostic system of claim 3 .

5. the threshold value when the state identification means identifies the sky as the background of the trees is set higher than the threshold value when the sky is not identified; The diagnostic system according to claim 3 or 4.

6. the determination means further uses at least one of the planting date and pruning history of the tree to determine whether or not work to remove at least a portion of the tree is necessary. The diagnostic system according to claim 3 or 4.

7. the object is a tree, and the state of the object is the thickness of a branch of the tree; the determination means determines that work to remove at least a portion of the tree is necessary when the coverage rate of the facility by the tree is equal to or greater than a threshold set according to the thickness of the tree branches; The diagnostic system of claim 2 .

8. The threshold value when the thickness of the tree branch is less than the standard is set higher than the threshold value when the thickness of the tree branch is equal to or greater than the standard. The diagnostic system of claim 7.

9. Obtaining an image of the road Detecting equipment on the road that provides information to pedestrians based on the image; Estimating a coverage rate of the facility by objects that appear to overlap in the image; Identifying the state of the objects that appear to overlap; determining whether or not an operation to remove at least a portion of the object is necessary based on the estimated coverage rate of the facility and the identified state of the object; outputting the result of the determination; Diagnostic methods.

10. A process of acquiring an image of a road; A process of detecting equipment that provides information to pedestrians on a road based on the image; a process of estimating a coverage rate of the facility by objects that appear to overlap in the image; A process for identifying a state of the objects that appear to overlap; a process of determining whether or not an operation to remove at least a portion of the object is necessary based on the estimated coverage rate of the equipment and the identified state of the object; A process of outputting the result of the determination. A diagnostic program that causes a computer to run