Class inference system, tree species map generation system, class inference method, and tree species map generation method

Abstract

To infer a class of an object, such as a tree, easier than before.SOLUTION: In a learning phase, feature vectors F1, ..., FN of an object image 62G of a tree for each of multiple classes are calculated using a feature extraction network 51. In an inference phase, a feature vector FA of an object image 66 of a target tree which is to be subjected to inference is calculated using the feature extraction network 51. The class of the target tree is inferred based on the feature vector F1 and the feature vectors F1, ..., FN.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to techniques for classifying objects such as trees in a forest. [Background technology]

[0002] As described in Non-Patent Document 1, conventionally, methods have been proposed in which forest information is acquired using special hardware such as a hyperspectral camera, a multispectral camera, or a LiDAR (Light Detection and Ranging) sensor, and the types of trees growing in the forest are identified based on the acquired information.

[0003] However, since this hardware is expensive, this method increases costs. Therefore, a method of inferring tree species using AI (Artificial Intelligence) has been proposed.

[0004] For example, according to the method described in Patent Document 1, the contours of each of multiple objects present in a test area are identified based on a photograph of the test area taken from above or a first distribution that is the distribution of heights at each point in the test area and a second distribution that is the distribution of gradients at each point. For each identified contour, a label corresponding to the type of object present in the contour from among multiple predetermined types is assigned, and training data is generated for each contour, showing partial images of the portion surrounded by each contour in the photograph and the labels assigned to each contour. A trained model is generated using the partial images and labels shown in each training data as input and correct answer, respectively. The type of tree to be inferred is then inferred based on the generated trained model. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2020-91640 [Non-patent literature]

[0006] [Non-Patent Document 1] "Review of studies on tree species classification from remotely sensed data." Remote Sensing of Environment Sensing of Environment Sensing of Environment Sensing of Environment Sensing of Environment , 186 , 64 -87.,Fassnacht, FE, Latifi, H., Sterenczak, K., Modzelewska, A., Lefsky, M., Waser, LT, ... & Ghosh, A. Written in 2016 Summary of the Invention [Problem to be solved by the invention]

[0007] However, conventional AI methods such as those described in Patent Document 1 are unable to distinguish classes (types) that have not been trained. Therefore, an existing trained model generated based on training data acquired under a certain vegetation cannot be used in an area with a different vegetation.

[0008] One possible solution is to generate a new trained model based on the vegetation of the area, but this would require acquiring a huge amount of new training data, and would also require a high-performance computer for machine learning calculations.

[0009] In view of these problems, the present invention aims to make it easier than ever to build an AI that infers classes such as trees. [Means for solving the problem]

[0010] A class inference system according to one aspect of the present invention comprises: Grows in a specific region, Belonging to multiple classes tree canopy Represents the characteristics of the image First feature vector of, The image in question Pre-trained model By typing in a first means of acquiring; Grows in the specific region, the object of reasoning that is the subject of reasoning tree canopy Represents the characteristics of the image Second feature vector of, The image in question The trained model By typing in a second acquisition means for acquiring the inference target from among the plurality of classes; trees The class to which belongs is the first feature vector for each of the plurality of classes and The tree to be inferred The aforementioned Second feature vector and a reasoning means for making inferences based on the The trained model is a network that calculates the feature vector of an input image by convolution processing in a CNN (Convolutional Neural Network) generated by deep learning using the class and crown images of multiple trees as the objective variable and explanatory variable, respectively. A tree species map generation system according to one embodiment of the present invention is a tree species map generation system that generates a tree species map of an area having multiple regions and in which trees belonging to each of multiple classes grow, and includes a first acquisition means that acquires, for each of the multiple regions, a first feature vector representing the characteristics of an image of the crown of a tree belonging to each of the multiple classes that grows in the area by inputting the image into a trained model; a second acquisition means that acquires a second feature vector representing the characteristics of an image of the crown of each of multiple inference target trees that grow in the area by inputting the image into the trained model; an inference means that infers the class to which each of the multiple inference target trees belongs based on the first feature vector of each of the multiple classes in the area in which the inference target tree grows among the multiple regions and the second feature vector of the inference target tree; and a map generation means that generates the tree species map based on location information of each of the multiple inference target trees and the inference results by the inference means. [Effects of the Invention]

[0011] According to the present invention, it is possible to construct an AI that can infer classes such as trees more easily than before. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating an example of the overall configuration of a tree classification system. [Figure 2] FIG. 1 illustrates an example of a hardware configuration of a computer. [Figure 3] FIG. 2 is a diagram illustrating an example of a functional configuration of a computer. [Figure 4] FIG. 1 illustrates an example of a tree classification network. [Figure 5] 1A and 1B are diagrams showing examples of an aerial photograph and a canopy map, respectively. [Figure 6] FIG. 10 is a diagram illustrating an example of object data. [Figure 7] FIG. 10 is a diagram illustrating an example of an inference method. [Figure 8] FIG. 10 is a diagram illustrating an example of the distribution of feature points in a 2640-dimensional space. [Figure 9] FIG. 10 is a diagram illustrating an example of a tree classification map. [Figure 10]10 is a flowchart illustrating an example of the overall processing flow of a tree classification program. [Figure 11] FIG. 10 is a diagram showing an example of a correct / incorrect result. [Figure 12] FIG. 1 shows an example of a method for generalizing a feature extraction network and creating a national tree map. DETAILED DESCRIPTION OF THE INVENTION

[0013] [Overall structure] Fig. 1 is a diagram showing an example of the overall configuration of a tree classification system 3. Fig. 2 is a diagram showing an example of the hardware configuration of a computer 1. Fig. 3 is a diagram showing an example of the functional configuration of the computer 1.

[0014] As shown in Figure 1, the tree classification system 3 is composed of a computer 1 and a drone 2. The tree classification system 3 provides a service that uses AI (artificial intelligence) to classify trees that appear in aerial photographs.

[0015] Drone 2 is a UAV (Unmanned Aerial Vehicle) equipped with a digital camera, and is used to capture color photographs of the forest from above to form the basis of the learning data. Drone 2 may be a commercially available drone, such as the DJI PHANTOM 4.

[0016] As shown in FIG. 2, the computer 1 is composed of a processor 10, a RAM (Random Access Memory) 11, a ROM (Read Only Memory) 12, an auxiliary storage device 13, a network adapter 14, a keyboard 15, a pointing device 16, an input / output board 17, a touch panel display 18, and an audio output unit 19.

[0017] In addition to the operating system, various other programs are installed in the ROM 12 or the auxiliary storage device 13. In particular, in this embodiment, a tree classification program 4 (see FIG. 3) is installed. The auxiliary storage device 13 may be a solid state drive (SSD) or a hard disk.

[0018] The RAM 11 is the main memory of the computer 1. In addition to the operating system, programs such as the tree classification program 4 are loaded into the RAM 11.

[0019] The processor 10 executes a program loaded into the RAM 11. The processor 10 may be a GPU (Graphics Processing Unit) or a CPU (Central Processing Unit).

[0020] The network adapter 14 is a device for communicating with other devices such as the drone 2 using protocols such as TCP / IP (Transmission Control Protocol / Internet Protocol).

[0021] The keyboard 15 and the pointing device 16 are input devices that allow the operator to input commands, data, and the like.

[0022] The input / output board 17 communicates with the drone 2 via a wired or wireless connection. As the input / output board 17, for example, an input / output board conforming to USB (Universal Serial Bus) or Bluetooth is used.

[0023] The touch panel display 18 displays a screen for inputting commands or data, or a map generated by the processor 10, etc.

[0024] The audio output unit 19 is composed of an audio board, a speaker, etc., and outputs sounds such as warning sounds.

[0025] The tree classification program 4 is a computer program for realizing the canopy map generation unit 41, label assignment unit 42, feature vector calculation unit 43, training data storage unit 44, canopy detection unit 45, feature calculation unit 46, class inference unit 47, tree classification map generation unit 48, and tree classification map output unit 49 shown in Fig. 3. The tree classification program 4 can generate a classifier for inferring the class (type) of a tree, and infer the class of a target tree based on this classifier.

[0026] Below, we will explain the processing of the drone 2 and each part shown in Figure 3, broadly dividing it into a learning phase and an inference phase, using an example of tuning and using the tree classification system 3 to classify trees in a forest 80 in a certain area (site).

[0027] [Learning Phase] (1) Network and Data Fig. 4 is a diagram showing an example of a tree classification network 5. Fig. 5 is a diagram showing examples of an aerial photograph 60 and a canopy map 61. Fig. 6 is a diagram showing examples of object data 62.

[0028] A feature extraction network 51 is prepared in advance in the computer 1. The feature extraction network 51 is a part of the tree classification network 5 shown in FIG.

[0029] The tree classification network 5 is a CNN (Convolutional Neural Network) that infers which of N classes the tree to be classified belongs to based on an image of the tree's canopy, and is mainly composed of a feature extraction network 51 and a probability calculation network 52.

[0030] The feature extraction network 51 is a network having an input layer, a convolutional layer, a pooling layer, and an output layer, and calculates a feature vector F representing the features of an input image. The convolutional layers and pooling layers may be arranged alternately. In this embodiment, a 224 x 224 pixel RGB color image is input to the input layer, and a 2560-dimensional vector is output from the output layer as a feature vector through calculations in each layer.

[0031] The probability calculation network 52 is composed of an input layer, one or more fully connected layers, and an output layer. Softmax is used as the activation function for the output layer. When a feature vector F is input to the input layer, the probabilities p1, p2, ..., p3 of the target tree belonging to the first, second, ..., Nth classes are calculated by the calculations of each layer. N is output from the output layer.

[0032] The tree classification network 5 may be an existing network (trained model) trained based on images of the tree canopy of forests other than the forest 80. For example, it may be created using the method described in JP 2020-91640 A or a watershed segmentation method. According to the method described in JP 2020-91640 A, a computer 1 generates an orthophotograph using an aerial photograph 60, generates a DSM (Digital Surface Model) based on the orthophotograph, and generates a slope model based on the DSM. A canopy map 61 is then generated based on the red, green, and blue gradations of the orthophotograph, the DSM model, and the slope model.

[0033] The vegetation of the forest that the tree classification network 5 has learned may be different from the vegetation of the forest 80. In other words, a CNN feature extraction network generated for a forest other than the forest 80, regardless of the vegetation, can be used as the feature extraction network 51 in the computer 1.

[0034] The operator flies the drone 2 over the forest 80 and photographs a part or the whole of the forest 80 from a predetermined altitude, thereby obtaining an aerial photograph 60 as shown in FIG. 5(A). The aerial photograph 60 is then input into the computer 1. The aerial photograph 60 is an RGB color photograph.

[0035] The canopy map generator 41 (see FIG. 3) of the computer 1 then detects the positions and shapes of objects such as canopies that appear in the aerial photograph 60, and generates a canopy map 61 as shown in FIG. 5(B). In the canopy map 61, the position and shape of each object are represented as polygons 61A. The canopy map 61 can be created by a known method. For example, it can be created by the method described in JP 2020-91640 A.

[0036] The labeling unit 42 assigns labels corresponding to the classes of some of the objects to the images of the objects captured in the aerial photograph 60 as follows.

[0037] The labeling unit 42 displays the canopy map 61 on the touch panel display 18. Instead of the canopy map 61, the outlines of the polygons 61A may be displayed superimposed on the aerial photograph 60.

[0038] Here, the operator selects (samples) one to several polygons 61A representing the canopy for each class of trees growing in the forest 80. For example, if the classes of trees growing in the forest 80 are metasequoia, white pine, and Japanese cypress, the operator selects one to several polygons 61A representing metasequoia, one to several polygons 61A representing white pine, and one to several polygons 61A representing Japanese cypress. In addition, the operator selects one to several polygons 61A representing non-tree objects (for example, the ground). It is desirable to select the same number of polygons 61A for each class. That is, for example, it is desirable to select three polygons 61A each for metasequoia, white pine, Japanese cypress, and the ground.

[0039] The labeling unit 42 then extracts an image of an object corresponding to each selected polygon 61A from the aerial photograph 60 as an object image 62G, and assigns a label identifying the object to the object image 62G as a label 62L. For example, if the selected polygon 61A is a metasequoia, the labeling unit 42 extracts an image of an object corresponding to the polygon 61A from the aerial photograph 60 as an object image 62G, and assigns the metasequoia label to the object image 62G as a label 62L. Then, the labeling unit 42 generates object data 62 indicating the object image 62G and the label 62L.

[0040] Through the operation of the operator and the processing of the labeling unit 42, sample object data 64 for each class is generated as shown in FIG.

[0041] 6, the values ​​of the label 62L are "Class_1", "Class_2", etc., but they may also be type names such as "Red Oak", "Red Pine", "Other", etc. In this case, the operator can simply input each type name when selecting the polygon 61A.

[0042] (2) Learning Fig. 7 is a diagram showing an example of an inference method, Fig. 8 is a diagram showing an example of the distribution of feature points in a 2640-dimensional space, and Fig. 9 is a diagram showing an example of a tree classification map 67.

[0043] When the object data 62 is generated, the feature vector calculation unit 43 (see FIG. 3) calculates a feature vector F of each object image 62G of the object data 62 by inputting the object image 62G to the feature extraction network 51, as shown in FIG. 7. The object image 62G is input after being appropriately reduced in size to match the size of the input layer of the feature extraction network 51.

[0044] Then, the feature vector calculation unit 43 generates teacher data 63 by replacing the object image 62G in the object data 62 with the calculated feature vector F, and stores the teacher data 63 in the teacher data storage unit 44.

[0045] According to the processing of the canopy map generation unit 41 or the feature vector calculation unit 43, for example, if the operator selects five polygons 61A for each of 11 classes of objects (trees or other objects), 55 pieces of training data 63 are generated and stored in the training data storage unit 44.

[0046] The following description will be given taking as an example a case where P pieces of teacher data 63 are stored in the teacher data storage unit 44. The feature vectors F shown in the respective teacher data 63 are referred to as "feature vector F1", "feature vector F2", ..., "feature vector F P " is sometimes written to distinguish it from the "

[0047] These feature vectors F can be thought of as position vectors starting from the origin of 2640-dimensional space (Euclidean space). Hereinafter, the end points of feature vectors F in 2640-dimensional space will be referred to as "feature points." When the feature points of the feature vectors F of each object are plotted in 2640-dimensional space, the distribution of the features of the objects present in forest 80 appears, as shown in Figure 8(A).

[0048] [Inference Phase] The operator inputs an aerial photograph 65, which is a photograph of a part or the whole of a forest 80 taken by a drone 2 from a predetermined altitude, into the computer 1. The aerial photograph 65, like the aerial photograph 60, is an RGB color photograph.

[0049] When an aerial photograph 65 is input, the canopy detection unit 45 (see FIG. 3) detects the position and shape of an object such as a canopy that appears in the aerial photograph 65. The detection method is the same as the detection method used by the canopy map generation unit 41, and for example, the method described in JP 2020-91640 A is used.

[0050] When the position and shape of each object, such as a canopy, captured in the aerial photograph 65 is detected by the canopy detection unit 45, processing for inferring the class of each object is performed by the feature calculation unit 46 and the class inference unit 47. In the following, an example of inferring the class of a certain object α will be described.

[0051] The feature amount calculation unit 46 extracts an image of the object α as an object image 66 from the aerial photograph 65 based on the position and shape of the object α. Then, the object image 66 is input to the feature extraction network 51 to calculate the feature vector F of the object α. This feature extraction network 51 calculates the feature amounts F1, F2, ..., F P Hereinafter, the feature vector F calculated by the feature amount calculation unit 46 will be referred to as the "feature vector F A " should be written.

[0052] The class inference unit 47 calculates the feature vectors F (F1, F2, ..., F P ) feature vector F A The k feature points closest to the feature point F are selected using the k-nearest neighbor method. Then, labels 62L of the training data 63 corresponding to the feature vector F of each selected feature point are extracted, and the class indicated by the most labels 62L is inferred to be the class of the object α. The distance between two feature points is the Euclidean distance.

[0053] For example, as shown in Figure 8(B), the feature vector F A If the end point of is located at "x" and k is "3", then the feature vector F A The three closest end points to the end point of are the three points surrounded by the dotted line, and of these three points, the most common are end points of class_1. Therefore, the class inference unit 47 infers that class_1 is the class of object α.

[0054] The class inference unit 47 infers the class of each of the other objects whose positions and shapes have been determined by the canopy detection unit 45, using the k-nearest neighbor method, in the same way as for the object α.

[0055] The tree classification map generation unit 48 generates a tree classification map 67 as shown in Fig. 9 based on the class inferred by the class inference unit 47 for each object whose position and shape have been determined by the canopy detection unit 45. Specifically, the tree classification map 67 is generated by replacing the image of each object captured in the aerial photograph 65 with a polygon of a color corresponding to the class of each object.

[0056] The tree classification map output unit 49 then outputs the generated tree classification map 67 by displaying it on the touch panel display 18. Alternatively, the tree classification map 67 file may be transmitted to another computer via a communication line.

[0057] [Overall Processing Flow and Effects of This Embodiment] Fig. 10 is a flowchart illustrating an example of the overall processing flow by the tree classification program 4. Fig. 11 is a diagram showing an example of a correct / incorrect result.

[0058] Next, the overall flow of processing by the computer 1 will be described with reference to the flowchart in Fig. 10. The computer 1 executes processing based on the tree classification program 4 in the procedure shown in Fig. 10.

[0059] When an aerial photograph 60 such as that shown in Figure 5(A) is input, the computer 1 generates a canopy map 61 such as that shown in Figure 5(B) based on the aerial photograph 60 (#101 in Figure 10) and displays it (#102).

[0060] When the operator selects one or several polygons 61A corresponding to each class, the computer 1 generates object data 62 as shown in Fig. 6 by assigning the corresponding class label 62L to the original image (object image 62G) of each selected polygon 61A (#103). Then, the computer 1 calculates the feature vector F for each object image 62G (#104), and generates and stores training data 63 indicating the feature vector F and label 62L for each object image 62G (#105). The above processing completes the learning phase.

[0061] When the aerial photograph 65 is input, the computer 1 detects the position and shape of the objects shown in the aerial photograph 65 (#121) and calculates the feature vector F of the object image 66 of each object in the aerial photograph 65 (#122).

[0062] Furthermore, the computer 1 infers the class of each object in the aerial photograph 65 using the k-nearest neighbor method based on the feature vector F shown in each piece of training data 63 and the feature vector F calculated in step #122 (#123). Then, a tree classification map 67 such as that shown in Figure 9 is generated based on the inference results (#124) and output (#125).

[0063] According to this embodiment, the operator does not need to prepare a huge amount of training data as in the case of conventional deep learning, but only needs to prepare one or several pieces of object data 62 for each class (type) of trees growing in a certain area (site). Based on this object data 62, the computer 1 generates training data 63, completing preparations for inference. Therefore, according to this embodiment, it is possible to build an AI that infers classes such as trees more easily than before.

[0064] Here, an example of experimental results will be presented to further explain the effects of this embodiment. A CNN was generated by deep learning to classify nine types of trees (Japanese koshiabura, Mizume, beech, Japanese walnut, magnolia, adzuki pear, Japanese cherry, sugar maple, and horse chestnut) and non-tree objects (gaps) into one of 10 classes. The CNN was used to classify the objects to be inferred, and the actual classes and the inferred classes were tallied to obtain the true / false result_1 shown in FIG. 11.

[0065] On the other hand, in accordance with the method of this embodiment, a small amount of training data 63 (for example, 10 for each class) is generated using an existing feature extraction network 51, and the object to be inferred is classified using the k-nearest neighbor method based on this training data 63. The actual classes and the inferred classes are then tallied, and a correct / incorrect result _2 is obtained.

[0066] The kappa values ​​for Correct / Incorrect Results_1 and Correct / Incorrect Results_2 are 0.414 and 0.456, respectively. In other words, even with the k-nearest neighbor method using a set of training data63 of approximately 10 items per class acquired for each region (site), it is possible to achieve classification accuracy on the same level as that of CNN acquired using deep learning. Furthermore, the classification accuracy of some classes (sugar maple) can be significantly improved compared to deep learning.

[0067] [Modifications and Applications] The feature vector F of the object image 66 of the object to be inferred (for example, the feature vector F in FIG. 7) A ) feature points and feature vectors F1, F2, …, F P If the distances to all the feature points are equal to or greater than a predetermined distance, it is highly likely that the object does not belong to any class. In such a case, the computer 1 may infer the class of the object using a tree classification network 5 (see FIG. 4) without using the k-nearest neighbor method. That is, an object image 66 of the object is input to a feature extraction network 51, and probabilities p1, p2, ..., p are output from a probability calculation network 52. N The class corresponding to the highest probability among these may be inferred to be the class of the object.

[0068] In this embodiment, as explained in Fig. 8, the computer 1 infers the class of the target object by the k-nearest neighbor method in 2640 dimensions, but the inference may be performed after reducing the dimensions. For example, it is also possible to determine the most important elements (for example, 10) of the 2640 elements (components) of the feature vector F by the Sequential Forward Selection (SFS) method, and infer the class of the target object by the k-nearest neighbor method based on the feature quantities of the determined elements. That is, for example, for the feature vectors F1, F2, ..., F in Fig. 7, P are respectively calculated as low-dimensional vectors G1, G2, ..., G P and the feature vector F A is calculated by the SFS method as a low-dimensional vector G AThen, the low-dimensional vector G A feature points and feature vectors F1, F2, ..., F P The distance to each feature point is calculated and the class is inferred using the k-nearest neighbor method.

[0069] The most important elements may be determined based on the feature vector F of the image of the object to be inferred, i.e., object image 66, calculated in the inference phase (after step #122 in FIG. 10). Alternatively, the most important elements may be determined based on the feature vector F of any sample image, i.e., object image 62G, calculated in the learning phase (after #104).

[0070] Alternatively, important elements may be determined by a wrapper method other than the SFS method, such as the SBS (Sequential Backward Selection) method or a brute force method, or by a feature selection method other than the wrapper method.

[0071] FIG. 12 is a diagram showing an example of a method for generalizing the feature extraction network 51 and creating a tree map for the entire country.

[0072] The provider of the tree classification program 4 provides the tree classification program 4 to users throughout Japan and collects data from each user, thereby creating a tree species map for the entire country of Japan and improving the accuracy of the tree classification network 5 (see Figure 4). This mechanism will be explained below with reference to Figure 12.

[0073] The provider of the tree classification program 4 prepares in advance a server 31 having a tree classification network 5. Furthermore, the provider unifies in advance the identifiers (class codes or names) of the classes (types) of trees found in Japan, and prepares a tree list containing these identifiers.

[0074] Each time the server 31 receives a request from a user, it sends the feature extraction network 51 and tree list to the user's computer along with the tree classification program 4 (#131). The feature extraction network 51 is a neural network (general-purpose model) that is generally used in each user's region (site), and it is desirable that it be the latest version.

[0075] When the user receives the tree classification program 4 and the feature extraction network 51 on a computer, the user installs the tree classification program 4 and the feature extraction network 51 on the computer, which then functions as the computer 1.

[0076] The user uses the tree classification program 4, feature extraction network 51, and drone 2 to tune the tree classification program 4 to the forests in their area and create a tree classification map 67 (#132). Labeling follows the tree list. The user then sends the tree classification map 67 from their computer to the server 31 along with the location information of the photographed location and the object data 62 obtained during the creation process (#133).

[0077] Upon receiving the object data 62, tree classification map 67, and location information, the server 31 performs further learning (training) of the tree classification network 5 by deep learning using the object images 62G and labels 62L shown in the object data 62 as explanatory variables and target variables, respectively (#134). Part of the object data 62 may also be used as validation data or test data. Furthermore, the server 31 overlays the tree classification map 67 on a national map based on the location information (#135).

[0078] Through the above processing, the feature extraction network 51 is trained and a tree species map for all of Japan is created. The latest version of the feature extraction network 51 is then provided to each user. By repeating this series of processing, object data 62 is collected from various locations, and the feature extraction network 51 is trained to become a large-scale general-purpose model. Note that the tree classification network 51 may be subjected to deep learning so that a tree species map for the entire world, not just Japan, can be created and tree species can be classified worldwide.

[0079] In this embodiment, the k-nearest neighbor method is used to infer the class of an object such as a tree, but the k-nearest neighbor method may be modified as follows: The feature vector F (F1, F2, ...) of the object to be inferred is selected from the feature points of the feature vector F (F1, F2, ...) of each training data 63. A The k feature points closest to the feature points of are selected. Each selected feature point is classified into a class indicated by the label 62L of the training data 63 corresponding to each feature point. Then, for each class, the sum of the distances between each selected feature point and the feature points of the object to be inferred is calculated, and the class with the shortest sum is inferred to be the class of the object to be inferred. Alternatively, instead of votes, a larger score may be assigned to a feature point with a shorter distance, and the class with the largest sum of scores may be inferred to be the class of the object to be inferred.

[0080] Alternatively, instead of the k-nearest neighbor method, a learner can be generated using SVM (Support Vector Machine) or random forest, and the generated learner can be used to infer classes. Whether using SVM or random forest, feature vectors F1, F2, ..., F can be generated using the SFS method or similar. P We generate a learner by reducing the dimension of and then use the same method to generate the feature vector F A We can reduce the dimension of the model and infer classes.

[0081] 3 are integrated into computer 1, they may be distributed among multiple devices. For example, canopy map generation unit 41 and label assignment unit 42 may be provided in a first computer, feature vector calculation unit 43 may be provided in a second computer, training data storage unit 44 may be provided in a third computer, canopy detection unit 45 may be provided in a fourth computer, class inference unit 47 may be provided in a fifth computer, and tree classification map generation unit 48 and tree classification map output unit 49 may be provided in a sixth computer. Computer programs for realizing each function are installed in the first to sixth computers, and these computers cooperate with each other via communication lines to execute a series of learning and inference processes for generating tree classification map 67 from aerial photograph 60.

[0082] In this embodiment, the computer 1 is used to classify trees, but it may also be used to classify other objects. For example, it may be used to classify grassland flowers, shrubs, animal habitats (animal nests such as anthills and mole nests), or landforms. It may also be used to classify rocks (volcanic rocks, hypabyssal rocks, plutonic rocks, sedimentary rocks, etc.), or buildings.

[0083] Regardless of the type of classification, the computer 1 and the drone 2 basically need to perform the same processing as in this embodiment.

[0084] However, when classifying flowers and plants, the drone 2 collects aerial photographs 60, 65 by taking photographs at a lower altitude than when classifying trees. Because flowers and plants have a steep slope like the shape of a building, it is effective to use a slope model as in this embodiment, but because they are smaller than trees, it is desirable to take photographs at a lower altitude to match the size of the flowers and plants. The same applies when classifying shrubs and lake plants (e.g., reeds).

[0085] On the other hand, when classifying landforms (lakes, marshes, plains, hills, rivers), the drone 2 may collect aerial photographs 60, 65 by taking photographs at a higher altitude than when classifying trees. Alternatively, the drone 2 may collect aerial photographs 60, 65 with a lower resolution.

[0086] In addition, the configuration of the entire computer 1 or each part, the content of the processing, the order of processing, the data configuration, etc. may be modified as appropriate in accordance with the spirit of the present invention. [Explanation of symbols]

[0087] 1. Computers (learning devices) 3 Tree Classification System (Class Inference System) 31 Server (learning means) 4 Tree classification program (computer program) 42 Label assignment unit (first acquisition means) 46 Feature calculation unit (second acquisition means) 47 Class inference unit (inference means) 5. Tree Classification Network (CNN) 51 Feature Extraction Network (Trained Model) 63 Training data (data for inference) 62G Object Image 63 Training data (sample features) 66 Object image (image of the object to be inferred)

Claims

1. a first acquisition means for acquiring a first feature vector representing features of an image of a crown of a tree belonging to each of a plurality of classes that grows in a specific region by inputting the image into a trained model; a second acquisition means for acquiring a second feature vector representing features of an image of a crown of an inference target tree that is a target of inference and grows in the specific region by inputting the image into the trained model; an inference means for inferring a class to which the inference target tree belongs from among the plurality of classes, based on the first feature vector of each of the plurality of classes and the second feature vector of the inference target tree; and The trained model is a network of a part of a convolutional neural network (CNN) generated by deep learning using the classes of a plurality of trees and the images of tree crowns as objective variables and explanatory variables, respectively, that calculates the feature vectors of input images by convolution processing. A class inference system characterized by:

2. the inference means infers a class to which the inference target tree belongs among the plurality of classes by a k-nearest neighbor method; The class inference system of claim 1 .

3. the first acquisition means acquires, as the first feature vector of an image of a crown of a tree belonging to each of the plurality of classes, a first low-dimensional vector composed of each of a plurality of specific element values ​​of a feature vector calculated by inputting the image into the trained model; the second acquisition means acquires, as the second feature vector, a second low-dimensional vector consisting of each of the specific plurality of element values ​​of a feature vector calculated by inputting an image of a crown of the inference target tree into the trained model; 3. The class inference system according to claim 1 or claim 2.

4. The specific elements are obtained by performing a sequential forward selection (SFS) process on the feature vector calculated by the trained model.

4. The class inference system of claim 3.

5. a learning means for further training the CNN using images of tree crowns belonging to each of the plurality of classes and each of the tree classes as explanatory variables and objective variables; having 5. A class inference system according to claim 1.

6. A tree species map generation system for generating a tree species map for an area having multiple regions where trees belonging to each of multiple classes grow, comprising: a first acquisition means for acquiring, for each of the plurality of regions, a first feature vector representing features of an image of a crown of a tree growing in the region and belonging to each of the plurality of classes, by inputting the image into a trained model; a second acquisition means for acquiring a second feature vector representing features of an image of the crown of each of a plurality of inference target trees growing in the region by inputting the image into the trained model; an inference means for inferring a class to which each of the plurality of inference target trees belongs based on the first feature vector of each of the plurality of classes in an area where the inference target tree grows among the plurality of areas and the second feature vector of the inference target tree; a map generating means for generating the tree species map based on the position information of each of the plurality of inference target trees and the inference result by the inference means; A tree species map generation system comprising:

7. Obtaining a first feature vector representing features of an image of a crown of a tree belonging to each of a plurality of classes growing in a specific region by inputting the image into a trained model; A second feature vector representing features of an image of a crown of an inference target tree that is a target of inference and grows in the specific region is obtained by inputting the image into the trained model; inferring a class to which the inference target tree belongs from among the plurality of classes based on the first feature vector of each of the plurality of classes and the second feature vector of the inference target tree; As the trained model, a network of a part of a CNN (Convolutional Neural Network) that calculates feature vectors of input images by convolution processing is used, the part being generated by deep learning using the classes of a plurality of trees and the images of tree crowns as objective variables and explanatory variables, respectively. A class inference method characterized by:

8. A tree species map generation method for generating a tree species map for an area having multiple regions and in which trees belonging to multiple classes grow, comprising the steps of: For each of the plurality of regions, a first feature vector representing features of an image of a crown of a tree belonging to each of the plurality of classes growing in the region is obtained by inputting the image into a trained model; A second feature vector representing features of an image of a crown of each of a plurality of inference target trees growing in the region is obtained by inputting the image into the trained model; inferring a class to which each of the plurality of inference target trees belongs based on the first feature vector of each of the plurality of classes in a region where the inference target tree grows among the plurality of regions and the second feature vector of the inference target tree; generating the tree species map based on the position information and the inferred class of each of the plurality of inference target trees; A tree species map generation method characterized by:

Images