Tree inspection system, tree inspection method, and recording medium

Abstract

A tree inspection system according to the present disclosure comprises: an acquisition means for acquiring an image and position information of a capturing point of the image; a recognition means for recognizing a street tree from the image by image recognition; a measurement means for measuring at least one of a color of a region of the street tree in the image or deviation in the image of a region of the street tree; a determination means for determining soundness of the street tree on the basis of at least one of a distribution of the color or the deviation; and an output means for outputting the position information and a determination result of soundness of the street tree in association with each other.

Description

Tree inspection system, tree inspection method, and recording medium

[0001] The present disclosure relates to a tree inspection system and the like.

[0002] Roadside trees are sometimes planted along roads by road administrators. Road administrators and arborists visit the locations where the trees have been planted to inspect the health of the trees. The inspection results are then stored appropriately.

[0003] Patent Document 1 discloses a roadside tree management system used in surveying roadside trees. Using the roadside tree management system of Patent Document 1, a user inputs various information about roadside trees.

[0004] Japanese Patent Application Laid-Open No. 2019-095976

[0005] It is time-consuming to carry out a detailed survey of every single tree planted along roads, so if a simple inspection can be used to narrow down the number of roadside trees to be surveyed in detail, the amount of work required for the survey can be reduced.

[0006] One of the objects of the present disclosure is to provide a tree inspection system etc. that supports the investigation of the health of trees.

[0007] A tree inspection system in one aspect of the present disclosure includes an acquisition means for acquiring an image and location information of the location where the image was taken, a recognition means for recognizing street trees from the image by image recognition, a measurement means for measuring at least one of the color of the area of ​​the street tree in the image or the bias in the image of the area of ​​the street tree, a determination means for determining the health of the street tree based on at least one of the color distribution or the bias, and an output means for outputting the location information and the determination result of the health of the street tree in association with each other.

[0008] A tree inspection method in one aspect of the present disclosure acquires an image and location information of the location where the image was taken, recognizes street trees from the image using image recognition, measures at least one of the color of the area of ​​the street tree in the image or the bias in the area of ​​the street tree in the image, determines the health of the street tree based on at least one of the color distribution or the bias, and outputs the location information and the determination result of the health of the street tree in association with each other.

[0009] A program according to one aspect of the present disclosure causes a computer to execute a process of acquiring an image and location information of a location where the image was captured, recognizing street trees from the image by image recognition, measuring at least one of the color of an area of ​​the street trees in the image or a bias in the area of ​​the street trees in the image, determining the health of the street trees based on at least one of the color distribution and the bias, and outputting the location information and the determination result of the health of the street trees in association with each other. The program may be stored in a computer-readable non-transitory recording medium.

[0010] One example of the effect of the present disclosure is that it can assist in investigating the health of trees.

[0011] 1 is an explanatory diagram showing an example of connection between a tree inspection system and other devices. FIG. 2 is a block diagram showing an example of the configuration of a tree inspection system. FIG. 3 is a diagram showing an example of an image. FIG. 4 is a diagram showing an example of recognition results for roadside trees. FIG. 5 is a histogram showing an example of color distribution. FIG. 6 is a diagram showing an example of bias in roadside tree areas. FIG. 7 is a diagram showing an example of display of determination results. FIG. 8 is a flowchart showing an example of operation of the tree inspection system. FIG. 9 is a block diagram showing another example of the configuration of a tree inspection system. FIG. 10 is a diagram showing an example of an image. FIG. 11 is a flowchart showing an example of operation of the tree inspection system. FIG. 12 is a block diagram showing an example of the hardware configuration of a computer.

[0012] The tree inspection system according to the present disclosure is used by road administrators who manage roadside trees. Roadside trees are trees planted along roads, such as roadways and sidewalks. The road administrator may be, for example, a national or local government. After a simple inspection using the tree inspection system according to the present disclosure, the road administrator may visit the location where a roadside tree with poor health is planted and conduct an investigation. The road administrator may investigate the tree's vigor by, for example, examining the length of branches, dead or damaged branches and trunks, the condition of the pruned ends and decay, leaf size, and leaf color. Furthermore, the road administrator may investigate the tree's shape by, for example, examining the inclination or curvature of the trunk, defects or deadness, and the density and distribution of branches. Furthermore, after a simple inspection using the tree inspection system according to the present disclosure or after an investigation by the road administrator, the road administrator may request a detailed diagnosis of roadside trees with poor health from an arborist. Based on the results of the diagnosis, the arborist or other appropriate person may take appropriate action regarding the roadside tree. Treatments may include, for example, disease and pest prevention and treatment, pruning, fertilization, water management, soil improvement, and tree removal or transplanting.

[0013] This disclosure describes a tree inspection system that analyzes information about the color or bias of street trees from an image and uses the information to determine the health of the street trees. The bias of street trees in an image indicates the left-right balance of the trees. However, the information used by the tree inspection system according to this disclosure to determine the health of trees is not limited to the color and bias of street trees. For example, the tree inspection system may analyze information other than the color and bias of street trees from an image and use it for the determination.

[0014] [First embodiment] An example of a connection between a tree inspection system 100 and other devices according to the present disclosure will be described using Figure 1. The tree inspection system 100 is connected to other devices via a communication network 30, either wired or wirelessly. The tree inspection system 100 is connected to, for example, a camera 10, an administrator terminal 20, and a storage 40. Note that the tree inspection system 100 does not necessarily need to communicate with the camera 10 and the administrator terminal 20. Therefore, the tree inspection system 100 only needs to be connected to the camera 10 and the administrator terminal 20 as needed.

[0015] The camera 10 photographs roadside trees along a road. The camera 10 may be mounted on a mobile object 11. The camera 10 may be realized, for example, by a drive recorder mounted on a vehicle and capturing images of the road and the environment around the road. The drive recorder continuously captures images of at least one of the front and rear of the vehicle while the vehicle is traveling on the road. However, the camera 10 may also be installed so as to capture images above or to the sides of the vehicle. The camera 10 may also be mounted on various types of mobile objects 11. For example, the camera 10 may be mounted on other mobile objects 11 such as a bicycle or a drone. The camera 10 may also be carried by a person traveling on the road.

[0016] The photography data including the images captured by the camera 10 is stored in the storage 40. The camera 10 transmits the photography data including the images to, for example, the storage 40 or the tree inspection system 100. The photography data further includes location information of the point where the image was captured. The location information is acquired using, for example, a Global Navigation Satellite System (GNSS) or a Global Positioning System (GPS). The location information is represented by, for example, latitude and longitude or a position on a map. The photography data may further include the date and time when the image was captured.

[0017] The administrator terminal 20 presents information to the road administrator. The type of the administrator terminal 20 is not particularly limited, and may be a smartphone, a tablet terminal, a PC (Personal Computer), etc. The administrator terminal 20 accesses the storage 40, for example, and displays the information stored in the storage 40.

[0018] The storage 40 stores image data including images captured by the camera 10. The storage 40 may store, together with the images, the results of assessments of the health of roadside trees made by the tree inspection system 100. The storage 40 may store, for each point on the map, images and the results of assessments of the health of roadside trees in association with each other.

[0019] The storage 40 may store images of the same street tree taken at multiple points in time, and the results of assessments made by the tree inspection system 100 based on the images taken at the multiple points in time, in association with each other. It can be determined that the trees are the same if the location information of the locations where the images were taken matches. A street tree ID (Identifier) ​​that uniquely identifies the street tree may be assigned to each street tree. Furthermore, the storage 40 may store, for each street tree ID, whether or not action has been taken. For example, the storage 40 may register the fact that an arborist has taken action on an unhealthy street tree, the date and time of the action, or the content of the action.

[0020] An example configuration of the tree inspection system 100 according to the present disclosure will be described using Figure 2. The tree inspection system 100 includes an acquisition unit 101, a recognition unit 102, a measurement unit 103, a determination unit 104, and an output unit 105. Note that some of the functions of the output unit 105 may be implemented by the administrator terminal 20.

[0021] The acquisition unit 101 acquires an image and location information of the location where the image was captured. In one example, the acquisition unit 101 acquires, from the storage 40, image data including an image captured by the camera 10 and location information of the image. Alternatively, the acquisition unit 101 may acquire, from the camera 10, image data including an image and location information.

[0022] The acquisition unit 101 may acquire, as the image, a video continuously captured by the camera 10 while the moving object 11 is moving. The acquisition unit 101 may acquire images of a section of a road selected by a user who is a road administrator.

[0023] Fig. 3 is a diagram showing an example of an image acquired by the acquisition unit 101. The image in Fig. 3 is captured, for example, by the camera 10, which is a drive recorder, capturing an image of the area ahead of the mobile object 11 while the mobile object 11 is traveling. In the image in Fig. 3, the entire roadside tree, including its trunk and branches, is captured from the side of the roadside tree. Therefore, in the process described below, the tree inspection system 100 can measure the condition of the tree's trunk and leaves from the image.

[0024] The recognition unit 102 recognizes street trees from the image acquired by the acquisition unit 101 through image recognition. The recognition unit 102 may recognize street trees using a machine learning model. The model is, for example, a model that has learned the relationship between an input image of street trees and correct labels assigned to areas of street trees in the input image. When an image is input, the model outputs a result of recognizing the areas of street trees that appear in the image. For example, the model outputs a rectangle that encloses the area of ​​the street trees as a result of recognizing the street trees. Furthermore, the model may output a result of determining for each pixel whether or not the pixel represents a street tree as a result of recognizing the street trees.

[0025] An example of the recognition result of roadside trees from the image of Fig. 3 is shown in Fig. 4. The recognition unit 102 may output the recognition result of roadside trees as a rectangular area R1, as shown in Fig. 4. In Fig. 4, the rectangular area R1 including the recognized roadside trees has sides parallel to the top, bottom, left, and right sides of the image.

[0026] The recognition unit 102 may output the results of recognizing road trees as rectangles with the smallest area that touch the road trees in the image. To output such rectangles, the recognition unit 102 uses, for example, the results of recognizing the outlines of road trees from the recognition results using a model. Here, the recognition results using a model may be rectangles that enclose the area of ​​the road trees, or may be the results of determining for each pixel whether or not a pixel represents a road tree. For example, the recognition unit 102 recognizes the outer outlines of road trees from the area of ​​the recognition results using the model. Then, the recognition unit 102 outputs the rectangle that touches the outer outline of the road tree and has the smallest area. The rectangle with the smallest area may be tilted with respect to the top, bottom, left, and right sides of the image.

[0027] The recognition unit 102 may distinguish between the trunk, branches, leaves, and flowers of roadside trees and recognize them. In this case, the recognition unit 102 uses, for example, a model trained using images in which the trunk, branches, etc. are labeled. The model may output, for example, a rectangle enclosing the trunk area and a rectangle enclosing the branch and leaf areas. The model may also output the result of determining for each pixel whether the pixel represents a trunk, branch, leaf, or flower. This allows the tree inspection system 100 to separately measure the condition of the tree trunk and the condition of the branches and leaves in the processing described below.

[0028] The recognition unit 102 may distinguish and recognize various types of street trees. The types of street trees may be classified by the height of the tree. For example, street trees may be classified as tall trees, medium-sized trees, shrubs, etc. based on their position and size in an image in which the entire tree is captured. If, among consecutively captured images, a street tree is recognized as a whole near the center of an image captured from a distance and the upper part or other portions are missing on the left and right sides of an image captured from a closer distance, the street tree is classified as a tall tree. Furthermore, the types of street trees may be classified in more detail by tree species, including ginkgo, cherry, azalea, etc. In this case, the recognition unit 102 uses, for example, a model that has learned the relationship between input images of various types of street trees and the correct tree species labels assigned to the images.

[0029] The measurement unit 103 measures at least one of the color of the roadside tree region in the image or the bias in the image of the roadside tree region. The measurement unit 103 may measure both the color of the roadside tree region and the bias in the image. The measurement unit 103 measures the color or the bias for the region recognized as a roadside tree region by the recognition unit 102.

[0030] The measurement of color by the measurement unit 103 will now be described. Measuring color means obtaining information about the color brightness of each pixel. The color of a pixel can be expressed by RGB values, which indicate the brightness of the three RGB (red, green, and blue) components. Therefore, the measurement unit 103 may measure color by obtaining the RGB values ​​of each pixel in the roadside tree area. The brightness of each RGB component is generally expressed in the range from 0 to 255. For a given pixel, if the brightness of the G component is significantly higher than the R and B components, the pixel appears green. Note that the color of each pixel may be expressed by values ​​other than RGB values, such as by using the HSV (hue, saturation, and brightness) color model.

[0031] By measuring color, the measurement unit 103 can obtain information about the color distribution. FIG. 5 shows an example of color distribution in an image. The histogram in FIG. 5 shows the distribution of brightness of the G component of a certain image. Since FIG. 5 has a peak on the right side, it indicates that there are many pixels in the image with high brightness in the G component. As with the G component, histograms of the R and B components of an image can also be plotted in FIG. 5. If the distribution of the R and B components is biased toward lower brightness, it is assumed that there are many green pixels in the image.

[0032] Note that the method of expressing the color distribution is not limited to the histogram for each component shown in FIG. 5 . The color distribution can also be expressed by counting the number of pixels of each color in a color space. Specifically, for example, the measurement unit 103 obtains color distribution information using a palette obtained by subtracting the RGB color space. The measurement unit 103 replaces the RGB values ​​of each pixel in the image with the closest color from the palette. The closeness of colors is expressed by the distance in the color space. Then, the measurement unit 103 obtains a distribution of the number of pixels for each color in the palette, such as green or orange.

[0033] The color distribution of a roadside tree area can also be represented by statistical values ​​obtained by statistically processing the RGB values ​​of each pixel in the area. For example, the measurement unit 103 calculates the average brightness of the R, G, and B components of each pixel. This allows the measurement unit 103 to calculate the average RGB value of the area. If there are many green pixels in the roadside tree area, the average RGB value will have a high brightness for the G component and low brightness for the R and B components.

[0034] When the recognition unit 102 outputs the area of ​​the roadside tree as a rectangular area, the measurement unit 103 measures, for example, the color of the pixels included in the rectangle. When the recognition unit 102 outputs the result of recognizing the area of ​​the roadside tree for each pixel, the measurement unit 103 measures the color of the recognized pixel. The recognition unit 102 may measure the color of the pixels in the areas of the branches, leaves, and flowers excluding the area of ​​the trunk.

[0035] Next, the measurement of bias by the measurement unit 103 will be described. The bias in an image of a roadside tree area is the left-right balance of the roadside trees in the image. The measurement unit 103 may measure the bias for the entire tree shown in the image. In this case, the measurement unit 103 can measure the left-right balance of the entire tree. Alternatively, the measurement unit 103 may measure the bias for the area of ​​the roadside tree trunk excluding the area of ​​the branches and leaves. This allows the measurement unit 103 to measure the angle of a fallen trunk or a trunk that grows at an angle. The bias can be measured by measuring the tilt or shape of a rectangle that encloses the roadside tree area. The bias in an image of a roadside tree area can also be measured by measuring the left-right distribution of pixels that represent the roadside trees included in the roadside tree area divided left and right by a straight line. The measurement method may be selected as appropriate, and a combination of multiple methods may be used.

[0036] Measurement of bias in an image of a roadside tree area will be described using Figure 6. Rectangle R1 is an example output of an area detected as a roadside tree area by the recognition unit 102. Rectangle R1 has sides parallel to the sides of the image acquired by the acquisition unit 101. Rectangle R2 is a diagram showing an example of a rectangle with the smallest area that contacts a roadside tree. When a tilted roadside tree is photographed, rectangle R2 with a different tilt from rectangle R1 may be recognized.

[0037] A case will be described in which the measurement unit 103 measures the tilt of a rectangle as the deviation of the roadside tree area. The measurement unit 103 may measure the tilt of a rectangle R2, which has the smallest area and is in contact with the roadside tree, as the deviation of the roadside tree area. Here, it is assumed that the camera 10 is attached horizontally to the ground and captures a horizontal image. In this case, the sides of the image can be used as the reference for the tilt. For example, the tilt of the rectangle R2 can be expressed by the angle formed between the left and right sides of the rectangle R2 and the left and right sides of the image, with the angle parallel to the left and right sides of the image being 0 degrees, using the left and right sides of the image as the reference. The larger the angle, the more the roadside tree area is tilted. The tilt of the rectangle R2 can also be expressed by the angle formed between the left and right sides of the rectangle R2 and the bottom side of the image. The further the angle is from 90 degrees, the more the roadside tree area is tilted.

[0038] A case will be described in which the measurement unit 103 measures the shape of a rectangle as the bias of the roadside tree area. The measurement unit 103 may measure the aspect ratio of a rectangle R1 having sides parallel to the sides of the image as the bias of the roadside tree area. The more biased the roadside tree area is, the longer the width of the rectangle R1 and the shorter the height.

[0039] A case will be described in which the measurement unit 103 measures the left-right distribution of pixels representing street trees divided by a straight line as the bias of the street tree area. In this case, the measurement unit 103 uses the result of the recognition unit 102 recognizing whether or not each pixel represents a street tree. The measurement unit 103 may divide the street tree area into left and right halves by a straight line passing through the center of the street tree trunk area. Alternatively, the measurement unit 103 may divide the street tree area into left and right halves by a straight line connecting the centers of the top and bottom sides of a rectangle that surrounds the street tree and whose sides are parallel to the sides of the image. A tree that is symmetrical and not biased has approximately the same number of pixels on the left and right. If the street tree area is biased, the number of pixels on either the left or right side will be larger.

[0040] The determining unit 104 determines the health of street trees based on at least one of the color distribution and the bias of the street tree area measured by the measuring unit 103. The determining unit 104 may determine the health of street trees in accordance with criteria that define the relationship between the health of at least one of the color distribution of the street tree area or the bias of the street tree area. The determining unit 104 may also determine the health of street trees in accordance with criteria that define the relationship between the health of both the color distribution and the bias of the street tree area. The determining unit 104 may determine the health of street trees using a binary value, indicating whether the street tree is healthy or unhealthy. The determining unit 104 may also represent the health of street trees in multiple stages.

[0041] The following describes a case in which the determination unit 104 determines the health of a roadside tree based on the color distribution of the roadside tree area. Healthy trees are expected to have lush green leaves and many pixels representing green leaves in the image. Therefore, the determination unit 104 uses, for example, the distribution of the luminance of the green component of each pixel among the RGB values ​​of each pixel in the roadside tree area. The determination unit 104 determines that the health of the recognized roadside tree is higher the more pixels with a high luminance of the green component are in accordance with a predetermined criterion. The determination unit 104 may also determine that the health of the recognized roadside tree is higher the greater the ratio of the number of pixels whose luminance of the green component is greater than a threshold to the total number of pixels in the roadside tree area. When the luminance of the green component is expressed as a range from 0 to 255, the threshold may be appropriately set, such as 200. The determination unit 104 may also determine the health of the roadside tree based on the distribution of the luminance of the red and blue components. The thresholds for the red and blue components are appropriately set.

[0042] The determination unit 104 may determine the tree health using the proportion of pixels of a predetermined color contained in the tree region as an index representing the color distribution. The colors used for the determination may be determined as appropriate. For example, if the number of green pixels is less than a predetermined standard, the determination unit 104 determines that the tree health is low because it is assumed that there are many dead branches. Also, for example, if the number of orange or brown pixels is more than a predetermined standard, the determination unit 104 determines that the tree health is low because it is assumed that there are many dead leaves and dead branches. If the number of blue pixels is more than a predetermined standard, the determination unit 104 determines that the tree health is low because it is assumed that there are few leaves and many pixels representing buildings and sky between the branches.

[0043] The determination unit 104 may determine the health of a roadside tree by using a statistical value obtained by statistically processing the RGB values ​​of each pixel in the roadside tree area as an index representing the color distribution. For example, if the similarity between a statistical value such as the average RGB value and the RGB value representing the color green is higher than a predetermined standard, the determination unit 104 determines that the roadside tree is in good health. The similarity to the color green is represented, for example, by the proximity of the distance in color space.

[0044] Next, a case where the determination unit 104 determines the health of a street tree based on the bias in the image of the street tree area will be described. Trees may lean due to damage to roots, branches, or leaves caused by disease or pests. Trees may also lean due to wind or soil collapse, which places environmental stress on the trees. Trees that are biased to the left or right are unstable and therefore at risk of falling. Therefore, the more biased a street tree is, the lower the health of the street tree is expected to be. For example, the determination unit 104 determines, according to a predetermined standard, that the smaller the bias in the image of the street tree area, the higher the health of the tree.

[0045] The above describes methods for determining the health of street trees based on both color distribution and the unevenness of street tree areas. The determination unit 104 may determine the health of a street tree by combining a criterion related to color distribution and a criterion related to the unevenness of street tree areas. The health of a street tree may be expressed by a combination of the vitality of both tree vigor and tree shape. Vitality is an index that represents the growth potential and health state of a tree. Therefore, the determination unit 104 may determine the vitality of tree vigor using the criterion related to color distribution and the vitality of tree shape using the criterion related to the unevenness of street tree areas. For example, a vitality of 1 indicates the highest health, and a vitality of 5 indicates the lowest health. The determination unit 104 determines that a street tree is healthy when both the vitality of tree vigor and tree shape are 1 or 2. The determination unit 104 determines that the health of a street tree is in a state where there is a possibility of damage requiring attention when either the vitality of tree vigor or tree shape is 3. The determining unit 104 determines that the tree is in a state where there is a possibility of significant damage if either the tree vigor or the tree shape vitality index is 4. The determining unit 104 may determine that the tree is unhealthy if either the tree vigor or the tree shape vitality index is 5.

[0046] The criteria used by the determination unit 104 for determination may be determined for each type of tree. For example, the color and quantity of leaves that indicate a healthy state may differ depending on the tree. Therefore, the determination unit 104 determines the healthiness using different criteria related to color distribution depending on the type of tree recognized by the recognition unit 102. Furthermore, because trees have different shapes, the degree of bias in the tree area in the image may differ. Therefore, the determination unit 104 determines the healthiness using different criteria related to bias in the roadside tree area depending on the type of tree recognized by the recognition unit 102.

[0047] The criteria used by the determining unit 104 for determination may be determined for each season. For example, the color distribution of a healthy state of a tree that loses leaves or blooms varies depending on the season. The shape of the tree area changes as leaves fall, and the degree of bias in an image of a healthy street tree may vary. Therefore, the determining unit 104 determines the healthiness using different criteria for the color distribution or bias in the area of ​​the street tree depending on the shooting date included in the shooting data acquired by the acquiring unit 101.

[0048] The above describes a case in which the determination unit 104 determines road tree health according to predetermined criteria. The determination unit 104 may determine road tree health using a trained model that has undergone machine learning to learn the relationship between the color distribution of road tree regions, the bias of road tree regions, and the correct label for road tree health. The model outputs a road tree health assessment result when the color distribution of road tree regions and the bias of road tree regions are input. Here, the method of representing the color distribution is appropriately selected, including the examples described above, such as the percentage of pixels with a green component brightness higher than a threshold. The method of representing the road tree regions is appropriately selected, including the examples described above, such as the angle of inclination of a rectangle with the smallest area that is adjacent to the road tree. The model may be a model that has learned the relationship between road tree health and other data, such as tree type or season, in addition to the color distribution and bias of road tree regions. Note that the model that has learned the relationship with other data may be a model that has learned the relationship between road tree health and either the color distribution or bias of road tree regions.

[0049] The output unit 105 outputs the location information of the shooting location of the image acquired by the acquisition unit 101 and the determination result of the health of the street trees determined by the determination unit 104 in association with each other. The output unit 105 may register the location information and the determination result in a database. For example, the output unit 105 may store the location information and the determination result in the storage 40. The output unit 105 may also display the determination result on the administrator terminal 20.

[0050] FIG. 7 is a diagram showing an example of a display screen of the determination result that the output unit 105 displays on the administrator terminal 20. In FIG. 7 , the determination result of the health of the street tree indicates that the street tree is "unhealthy." The output unit 105 may output the image and the image capture date acquired by the acquisition unit 101 in association with the determination result. In FIG. 7 , the image capture date is displayed as the "simple diagnosis date." The output unit 105 may output the recognition result of the street tree by the recognition unit 102 in association with the determination result. In FIG. 7 , a frame indicating the recognition result of the street tree by the recognition unit 102 is superimposed on the image. If the determination unit 104 determines the vitality level of the tree vigor from the color distribution and the vitality level of the tree shape from the bias of the street tree area, the output unit 105 may output the vitality level of the tree vigor and the vitality level of the tree shape. In FIG. 7 , it is displayed that the vitality level of the tree vigor is 3 and the vitality level of the tree shape is 2. The output unit 105 may output the vitality of the tree vigor and the vitality of the tree shape as the determination result regarding the health of the roadside trees.

[0051] The output unit 105 may output the determination result for the roadside tree selected by the user. For example, the output unit 105 may display the display screen of Fig. 7 for the roadside tree that exists at the point selected on the map.

[0052] The output unit 105 may display the determination result on a map using an icon indicating the position of the location where the image of the street tree was taken. For example, the output unit 105 may display icons on the map that have different appearances depending on the level of healthiness. The output unit 105 may display a darker icon or a larger icon the lower the healthiness. Furthermore, the output unit 105 may display the determination result by displaying on the map the positions of street trees whose healthiness is lower than a standard specified by the user. In this case, the output unit 105 may display the positions of each street tree using an icon with the same appearance.

[0053] 8 is a diagram showing an example of a display screen of the determination results that the output unit 105 displays on the administrator terminal 20. In Fig. 8, the locations of street trees that have been determined to be unhealthy are displayed on a map. If the user specifies that other health conditions be displayed, the output unit 105 displays the locations of the selected healthy street trees.

[0054] When a triangular icon is selected in Fig. 8, the output unit 105 may display a display screen such as that shown in Fig. 7 regarding roadside trees located at the selected point. As shown in Fig. 8, the output unit 105 may display the determination result on a map using a heat map indicating areas with many roadside trees in poor health. Areas with many roadside trees in poor health are indicated by a darker color. In Fig. 8, the user may select whether or not to display the heat map.

[0055] The output unit 105 may display on the map the type of roadside tree specified by the user, using the information on the type of roadside tree recognized by the recognition unit 102. Furthermore, the output unit 105 may display on the map different icons depending on the type of tree.

[0056] The output unit 105 may display the position of the photographing location on a map based on the assessment result and the priority of addressing the roadside tree, which is based on the traffic volume of the road at the photographing location where the street tree was photographed. The lower the health level indicated by the assessment result and the higher the traffic volume, the higher the priority is set. The output unit 105 may display the positions of a predetermined number of roadside trees on the map in descending order of priority. Note that the number of roadside trees to be displayed may be set based on information other than the priority of addressing and traffic volume.

[0057] The output unit 105 may output the location information of the image capture location and the health assessment result in a table format. For example, the output unit 105 may output the assessment result in CSV (Comma Separated Values) format. The data output by the output unit 105 may include a street tree ID, an image, or information identifying the image, in addition to the location information and the health assessment result. The street tree ID is identified by the determination unit 104 by comparing the information of the image capture location with the location information stored in the storage 40.

[0058] The output unit 105 may output the determination results in descending order of priority based on the determination results and the priority of measures based on the amount of traffic on the road. Note that the priority of measures may be set based on information other than traffic volume.

[0059] The output unit 105 may display the determination results in a table format, sorting the road tree types. This allows the road administrator to consider measures for each road tree type. The display order for each road tree type may be set by the user.

[0060] An example of the operation of the tree inspection system 100 according to the present disclosure will be described with reference to Fig. 9. The tree inspection system 100 may start the process of Fig. 9 when an image is collected in the storage 40 or at a predetermined time each month.

[0061] In step S11, the acquisition unit 101 acquires an image and location information of the location where the image was captured. In step S12, the recognition unit 102 recognizes street trees from the image acquired by the acquisition unit 101 through image recognition. In step S13, the measurement unit 103 measures at least one of the color of the street tree area in the image or the bias in the image of the street tree area. In step S14, the determination unit 104 uses the measurement results by the measurement unit 103 to determine the health of the street trees based on at least one of the color distribution or bias in the street tree area. In step S15, the output unit 105 outputs the location information of the image capture location and the determination result of the health of the street trees in association with each other.

[0062] With the above, the tree inspection system 100 ends the processing of Figure 9.

[0063] According to one embodiment, the determination unit 104 determines the health of street trees based on at least one of the color distribution and bias of the area of ​​the street trees measured from the image. The output unit 105 then outputs the location information of the image capture location and the determination result of the health of the street trees in association with each other. By viewing the output result of the output unit 105, road managers can narrow down the street trees to be subject to detailed investigation and diagnosis. Therefore, the tree inspection system 100 can reduce the effort required for investigation and support the investigation of the health of street trees.

[0064] According to one embodiment, the acquisition unit 101 acquires images captured by the camera 10 mounted on the vehicle. The determination unit 104 then determines the health of roadside trees using information measured from the images. Therefore, road managers can obtain the results of roadside tree health determinations simply by driving their vehicles. Therefore, the tree inspection system 100 allows for easy and regular inspection of roadside trees.

[0065] Second Embodiment An example configuration of a tree inspection system 200 according to the present disclosure will be described using Figure 10. The tree inspection system 200 differs from the tree inspection system 100 of Figure 2 in that it includes a target identification unit 106 and a depth estimation unit 107. Note that the tree inspection system 200 may include the depth estimation unit 107 as needed. Regarding the configuration of the tree inspection system 200 according to the second embodiment, a description of the same configuration as the configuration of the tree inspection system 100 according to the first embodiment will be omitted.

[0066] The target identification unit 106 uses the position or size in the image of the street tree recognized by the recognition unit 102 to identify a street tree to be subjected to health assessment from among the one or more recognized street trees. The position and size of an object vary depending on the distance from the camera 10. In the image of FIG. 11 , street tree T1 appears larger and to the left of street tree T2, which is farther from the camera 10. By identifying a street tree of a predetermined position or a predetermined size in the image, the target identification unit 106 can identify a street tree photographed from a distance appropriate for assessment from the camera 10. Here, the predetermined position or size is determined as appropriate so that a nearby street tree can be identified.

[0067] Trees captured in the distance may appear small, making it difficult to measure the color distribution. Trees captured in the image at a distance may not capture the entire tree, making it difficult to measure the color distribution and bias. The bias may vary depending on the distance between the camera 10 and the roadside tree, making it difficult for the determination unit 104 to make a determination based on a certain standard. Furthermore, when the output unit 105 correlates the position information with the determination result and outputs it, there is a possibility that the deviation in the position information of roadside trees captured from a distance may be large. Furthermore, distant trees captured in the image may not be roadside trees. Therefore, the target identification unit 106 has the advantage of identifying roadside trees captured from an appropriate distance.

[0068] A method for identifying a roadside tree to be determined based on its position in an image is described below. The target identification unit 106 can identify a roadside tree that is closer to the left or right edge of the image than a predetermined position and whose trunk is located lower than the predetermined position. This is because the closer the camera 10 is to the roadside tree, the closer and lower the tree may appear to the edge of the image. Note that the predetermined position is determined appropriately depending on the camera angle so that roadside trees that appear relatively close can be identified. The target identification unit 106 may identify roadside trees on both the left and right sides of the image. Alternatively, the target identification unit 106 may identify roadside trees that are closer to the edge of the lane on which the mobile object 11 is traveling than the predetermined position. In countries where left-hand traffic is practiced, the lane on which the mobile object 11 is traveling is set to the left. In this way, when roadside trees are installed on both the driving lane and the opposite lane, the target identification unit 106 can identify the roadside tree that is closer to the driving lane. The roadside trees on the opposite lane side may be identified if they appear on the driving lane side in an image captured while the vehicle is traveling in the opposite lane.

[0069] In one example, the object identification unit 106 identifies the street tree in the image that is closest to the lower corner of the lane on which the mobile object 11 is traveling. In countries where people drive on the left side of the road, the object identification unit 106 identifies the street tree that is closest to the bottom left corner in the image. In Fig. 11 , the object identification unit 106 compares street trees T1 and T2 and identifies the street tree T1 whose bottom edge is closest to the bottom left corner of the image.

[0070] The target identification unit 106 may identify street trees that are entirely visible. For example, the target identification unit 106 may identify street trees that are not adjacent to the top, bottom, left, or right edges of the image. The target identification unit 106 may also identify street trees that are entirely visible, using the results of determining whether each pixel represents a street tree. This allows the target identification unit 106 to exclude street trees that are partially missing or hidden.

[0071] Next, a method for identifying roadside trees based on their size in an image will be described. The target identification unit 106 identifies roadside trees whose size in an image is equal to or larger than a predetermined standard. In one example, the target identification unit 106 may compare the sizes of roadside trees in the image and identify the largest roadside tree.

[0072] The depth estimation unit 107 estimates the depth of an object in an image using a depth estimation technique that estimates the distance between the camera and the object from a single image. Note that when multiple cameras 10 are mounted on a single mobile object 11, the depth estimation unit 107 may calculate the distance from the camera to the object by restoring three-dimensional information from images captured by each camera 10. The multiple cameras 10 mounted on the mobile object 11 may be a stereo camera or two synchronized cameras.

[0073] If the tree inspection system 200 includes the depth estimation unit 107, the target identification unit 106 may further use the depth estimation result to identify, as the determination target, the street tree that is closest to the camera 10 that captured the image, among one or more recognized street trees. By using the depth, the target identification unit 106 can identify the closer street tree among street trees that appear at the same size.

[0074] An example of the operation of the tree inspection system 200 according to the present disclosure will be described using Fig. 12. With regard to the processing of the tree inspection system 200, a description of processing that is the same as the processing of the tree inspection system 100 described using Fig. 9 will be omitted.

[0075] After step S12, in step S21, the depth estimation unit 107 estimates the depth of the image. In step S22, the target identification unit 106 uses the position or size in the image of the street tree recognized by the recognition unit 102 to identify, among the multiple recognized street trees, the street tree closest to the camera 10 that captured the image as the target for health assessment. In step S22, the target identification unit 106 may further use the result of depth estimation by the depth estimation unit 107 to identify the street tree closest to the camera 10 as the target for health assessment.

[0076] In step S13, the measurement unit 103 measures the color or bias of the area of ​​the street tree identified by the target identification unit 106 as the target for evaluation. In step S14, the determination unit 104 determines the health of the street tree identified as the target for evaluation. Then, in step S15, the output unit 105 outputs the location information of the image capture location and the health determination result in association with each other. With this, the tree inspection system 200 ends the processing of FIG. 12.

[0077] Step S21 may be performed before step S12. Alternatively, step S21 may be omitted. Step S22 may be performed after step S13.

[0078] In one embodiment, the target identification unit 106 identifies a street tree to be assessed for healthiness from among one or more recognized street trees. The determination unit 104 then assesses the healthiness of the street tree identified as the assessment target. This allows the determination unit 104 to assess healthiness according to a set standard. Furthermore, the output unit 105 outputs accurate location information for the street tree.

[0079] In a modification of the tree inspection systems 100 and 200 according to the first and second embodiments, the recognition unit 102 may further recognize dead leaves lying on the ground by image recognition. The determination unit 104 may use the recognition result of the recognition unit 102 to determine that a roadside tree is unhealthy if there are dead leaves, or the more dead leaves there are, the less healthy the roadside tree is.

[0080] [Other Application Examples] Application examples of the tree inspection systems 100 and 200 according to the first and second embodiments are not limited to inspecting roadside trees. In addition to roadside trees, the tree inspection systems may also be used to inspect trees in orchards and mountains, for example.

[0081] [Hardware Configuration] In each of the above-described embodiments, each component of the tree inspection system 100, 200 represents a functional block. Some or all of the components of the tree inspection system 100, 200 may be realized by any combination of a computer 500 and a program.

[0082] Fig. 13 is a block diagram showing an example of the hardware configuration of a computer 500. Referring to Fig. 13, the computer 500 includes, for example, a processor 501, a read only memory (ROM) 502, a random access memory (RAM) 503, a program 504, a storage device 505, a drive device 507, a communication interface 508, an input device 509, an output device 510, an input / output interface 511, and a bus 512.

[0083] The processor 501 controls the entire computer 500. The processor 501 may be, for example, a CPU (Central Processing Unit). The number of processors 501 is not particularly limited, and there may be one or more processors 501.

[0084] The program 504 includes instructions for realizing each function of the tree inspection systems 100, 200. The program 504 is stored in advance in the ROM 502, RAM 503, or storage device 505. The processor 501 executes the instructions included in the program 504 to realize each function of the tree inspection systems 100, 200. The RAM 503 may also store data to be processed in each function of the tree inspection systems 100, 200.

[0085] The drive device 507 reads and writes data from and to the recording medium 506. The communication interface 508 provides an interface with a communication network. The input device 509 is, for example, a mouse or a keyboard, and receives information input from road administrators, etc. The output device 510 is, for example, a display, and outputs (displays) information to road administrators, etc. The input / output interface 511 provides an interface with peripheral devices. The bus 512 connects these hardware components. The program 504 may be supplied to the processor 501 via a communication network, or may be stored in advance on the recording medium 506, read by the drive device 507, and supplied to the processor 501.

[0086] It should be noted that the hardware configuration shown in FIG. 13 is an example, and other components may be added, or some components may not be included.

[0087] There are various variations in the method of realizing the tree inspection systems 100 and 200. For example, the tree inspection systems 100 and 200 may be realized by any combination of different computers and programs for each component. Furthermore, multiple components included in the tree inspection systems 100 and 200 may be realized by any combination of a single computer and program.

[0088] Furthermore, at least a part of the tree inspection systems 100 and 200 may be provided in a software as a service (SaaS) format. That is, at least a part of the functions for realizing the tree inspection systems 100 and 200 may be performed by software executed via a network.

[0089] Although the present disclosure has been described above with reference to the embodiments, the present disclosure is not limited to the above embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, the configurations in the respective embodiments can be combined with each other without departing from the scope of the present disclosure.

[0090] Some or all of the above embodiments may be described as, but are not limited to, the following supplementary notes.

[0091] [Supplementary Note 1] A tree inspection system comprising: an acquisition means for acquiring an image and location information of the location where the image was taken; a recognition means for recognizing street trees from the image by image recognition; a measurement means for measuring at least one of the color of the area of ​​the street tree in the image or the bias of the area of ​​the street tree in the image; a determination means for determining the health of the street tree based on at least one of the color distribution and the bias; and an output means for outputting the location information and the determination result of the health of the street tree in association with each other.

[0092] [Supplementary Note 2] The tree inspection system according to Supplementary Note 1, wherein the color distribution is a distribution of brightness for at least the green component of each pixel among the RGB values ​​of each pixel in the area of ​​the roadside trees.

[0093] [Supplementary Note 3] The tree inspection system according to Supplementary Note 2, wherein the determining means determines that the roadside tree is in good health as the proportion of pixels having a green component brightness greater than a threshold increases.

[0094] [Supplementary Note 4] The tree inspection system according to any one of Supplementary Notes 1 to 3, wherein the determining means determines the healthiness based on at least one of a proportion of green pixels or a proportion of orange pixels included in the area of ​​the roadside tree.

[0095] [Appendix 5] The tree inspection system described in any one of Appendices 1 to 4, wherein the determination means determines that the street tree is healthy when the similarity between the statistically processed RGB values ​​of each pixel in the area of ​​the street tree and the value representing green is higher than a reference value.

[0096] [Supplementary Note 6] The tree inspection system according to any one of Supplementary Notes 1 to 5, wherein the bias is a tilt in the image of a rectangle that has a minimum area and is in contact with the roadside tree in the image.

[0097] [Supplementary Note 7] The tree inspection system described in any one of Supplementary Notes 1 to 6, wherein the recognition means recognizes a rectangular area surrounding the street tree with sides parallel to sides of the image by object detection, and the bias is represented by an aspect ratio of the recognized rectangular area.

[0098] [Supplementary Note 8] The tree inspection system described in any one of Supplementary Notes 1 to 7, wherein the recognition means recognizes whether each pixel of the image represents a street tree, and the bias is represented by the left-right ratio of the number of pixels representing street trees when the area of ​​the street trees is divided by a straight line.

[0099] [Supplementary Note 9] The tree inspection system according to any one of Supplementary Notes 1 to 8, wherein the determining means determines the health of the street trees using a trained model that has been machine-learned to determine the relationship between at least one of the color distribution or the bias and the health of the street trees.

[0100] [Supplementary Note 10] The tree inspection system according to any one of Supplementary Notes 1 to 9, further comprising a target identification means for identifying a street tree to be subjected to the health assessment from among the one or more recognized street trees, using the position or size of the recognized street tree in the image.

[0101] [Supplementary Note 11] The tree inspection system according to Supplementary Note 10, further comprising a depth estimation means for estimating the depth of an object appearing in the image by depth estimation, wherein the target identification means uses the depth estimation result to identify, from among the one or more recognized street trees, the street tree that is closest to the camera that captured the image as the target for determination.

[0102] [Supplementary Note 12] The tree inspection system described in any one of Supplementary Notes 1 to 11, wherein the output means displays on a map the positions of the photographing points where the designated healthy street trees were photographed, based on the position information and the judgment results of the healthiness of the street trees.

[0103] [Supplementary Note 13] The tree inspection system according to Supplementary Note 12, wherein the output means displays the position of the photographing point on a map based on the result of the assessment of the health of the roadside tree and the priority of dealing with the roadside tree based on the amount of traffic on the road at the photographing point.

[0104] [Supplementary Note 14] A tree inspection system according to any one of Supplementary Notes 1 to 13.

[0105] [Supplementary Note 15] The tree inspection system according to any one of Supplementary Notes 1 to 14, wherein the output means outputs the location information and the determination results of the health of the roadside trees in a table format.

[0106] [Supplementary Note 16] The tree inspection system according to any one of Supplementary Notes 1 to 15, wherein the image is an image taken by a camera mounted on the vehicle while the vehicle is moving.

[0107] [Supplementary Note 17] A tree inspection method comprising: acquiring an image and location information of a location where the image was taken; recognizing street trees from the image by image recognition; measuring at least one of the color of an area of ​​the street tree in the image or a bias in the image of the area of ​​the street tree; determining the health of the street tree based on at least one of the color distribution and the bias; and outputting the location information and the determination result of the health of the street tree in association with each other.

[0108] [Supplementary Note 18] The tree inspection method according to Supplementary Note 17, wherein the color distribution is a distribution of brightness for at least the green component of each pixel among the RGB values ​​of each pixel in the area of ​​the roadside trees.

[0109] [Supplementary Note 19] A recording medium that non-temporarily records a program that causes a computer to execute the following processes: acquire an image and location information of a location where the image was taken; recognize street trees from the image by image recognition; measure at least one of the color of the area of ​​the street trees in the image or the bias of the area of ​​the street trees in the image; determine the health of the street trees based on at least one of the color distribution and the bias; and output the location information and the determination result of the health of the street trees in association with each other.

[0110] [Supplementary Note 20] The recording medium according to Supplementary Note 19, wherein the color distribution is a distribution of brightness for at least the green component of each pixel among the RGB values ​​of each pixel in the area of ​​the roadside trees.

[0111] Some or all of the configurations described in Supplementary Notes 2-16, which are dependent on Supplementary Note 1, may also be dependent on Supplementary Notes 17 and 19 in the same manner as Supplementary Notes 2-16. Not limited to Supplementary Notes 1, 17, and 19, some or all of the configurations described as Supplements may also be dependent on various hardware, software, various recording devices for recording software, or systems, within the scope of each of the above-mentioned embodiments.

[0112] 100, 200 Tree inspection system 101 Acquisition unit 102 Recognition unit 103 Measurement unit 104 Determination unit 105 Output unit 106 Target identification unit 107 Depth estimation unit 10 Camera 11 Mobile object 20 Administrator terminal 30 Communication network 40 Storage

Claims

1. An acquisition means for acquiring an image and position information of a shooting location of the image; a recognition means for recognizing street trees from the image by image recognition; a measurement means for measuring at least one of a color of a region of the street trees in the image or a bias in the image of the region of the street trees; a determination means for determining the soundness of the street trees based on at least one of the color distribution or the bias; and an output means for associating and outputting the position information and a determination result of the soundness of the street trees. A tree inspection system comprising the above components.

2. The tree inspection system according to claim 1, wherein the color distribution is a luminance distribution of at least a green component of each pixel among the RGB values of each pixel in the region of the street trees.

3. The tree inspection system according to claim 2, wherein the determination means determines that the street trees are sounder as the ratio of pixels having a luminance of the green component greater than a threshold value is larger.

4. The tree inspection system according to any one of claims 1 to 3, wherein the determination means determines the soundness based on at least one of a ratio of green pixels or a ratio of orange pixels included in the region of the street trees.

5. The tree inspection system according to any one of claims 1 to 4, wherein the determination means determines that the street trees are sound when the similarity between a value obtained by statistically processing the RGB values of each pixel in the region of the street trees and a value representing green is higher than a reference.

6. The tree inspection system according to any one of claims 1 to 5, wherein the bias is an inclination in the image of a rectangle having the smallest area in contact with the street trees in the image.

7. The tree inspection system according to any one of claims 1 to 6, wherein the recognition means recognizes a rectangular region surrounding the street trees with sides parallel to the sides of the image by object detection, and the bias is represented by the aspect ratio of the recognized rectangular region.

8. The tree inspection system according to any one of claims 1 to 7, wherein the recognition means recognizes whether each pixel of the image represents a street tree, and the bias is represented by the left-right ratio of the number of pixels representing the street trees when the region of the street trees is divided by a straight line.

9. The determination means determines the soundness of the street tree using a learned model obtained by machine learning the relationship between at least one of the color distribution or the bias and the soundness of the street tree. The tree inspection system according to any one of claims 1 to 8.

10. The tree inspection system according to any one of claims 1 to 9, further comprising target identification means for identifying, among the one or more recognized street trees, the street tree to be the target of the soundness determination using the position or size of the recognized street tree in the image.

11. The tree inspection system according to claim 10, further comprising depth estimation means for estimating the depth of an object shown in the image by depth estimation, and the target identification means uses the depth estimation result to identify, among the one or more recognized street trees, the street tree closest to the camera that captured the image as the determination target.

12. The tree inspection system according to any one of claims 1 to 11, wherein the output means displays the position of the shooting point where the street tree with the specified soundness was photographed on a map based on the position information and the determination result of the soundness of the street tree.

13. The tree inspection system according to claim 12, wherein the output means displays the position of the shooting point on a map based on the determination result of the soundness of the street tree and the priority of dealing with the street tree based on the traffic volume of the road at the shooting point.

14. The tree inspection system according to any one of claims 1 to 13.

15. The tree inspection system according to any one of claims 1 to 14, wherein the output means outputs the position information and the determination result of the soundness of the street tree in tabular form.

16. The tree inspection system according to any one of claims 1 to 15, wherein the image is an image captured by a camera mounted on the vehicle while the vehicle is moving.

17. A tree inspection method, which acquires an image and position information of a shooting point of the image, recognizes a street tree from the image by image recognition, measures at least one of the color of the area of the street tree in the image or the bias in the image of the area of the street tree, determines the soundness of the street tree based on at least one of the color distribution or the bias, and outputs by associating the position information with the determination result of the soundness of the street tree.

18. The method for inspecting trees according to claim 17, wherein the color distribution is a luminance distribution of at least the green component of each pixel among the RGB values of each pixel in the area of the street tree.

19. A recording medium that non-temporarily records a program for causing a computer to execute a process of: acquiring an image and position information of a shooting point of the image; recognizing a street tree from the image by image recognition; measuring at least one of a color in an area of the street tree in the image or a bias in the image in the area of the street tree; determining the soundness of the street tree based on at least one of the color distribution or the bias; and outputting by associating the position information with a determination result of the soundness of the street tree.

20. The recording medium according to claim 19, wherein the color distribution is a luminance distribution of at least the green component of each pixel among the RGB values of each pixel in the area of the street tree.

Images