Fine-grained city tree species classification method and device based on multi-modal data fusion

Abstract

The invention relates to the technical field of computer vision and laser scanning, in particular to a fine-grained city tree species classification method and device based on multi-modal data fusion, and the method comprises the steps: collecting the original point cloud of a street tree and a street view image of a target street; performing tree species information identification on the original point cloud of the border tree to construct a point cloud tree species classification data set; performing image cutting on the streetscape image according to the point cloud tree species classification data set to construct an image tree species data set; training a pre-constructed deep learning neural network by using the point cloud tree species classification data set and the image tree species data set until a preset requirement is met, and obtaining a final tree species classification model; and performing tree species classification on the to-be-classified urban street trees by using the final tree species classification model to obtain tree information of the to-be-classified urban street trees. Therefore, the problem that an existing image classification method based on deep learning is low in accuracy of automatically identifying the street tree species is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and laser scanning, in particular to a multi-modal data fusion fine-grained urban tree species classification method and device. BACKGROUND

[0002] As the core component of road greening, street trees can absorb carbon dioxide, purify air, regulate local climate, conserve water resources, and play an indispensable role in maintaining urban ecological systems. Street tree asset survey is the basis for urban ecological research and is of great significance for estimating urban regional carbon storage and conducting three-dimensional modeling of "digital city". Street tree resource inventory mainly includes obtaining three-dimensional structural information such as tree height, diameter at breast height, and crown width of urban trees, as well as tree species information. Traditional methods mainly rely on manual field measurement of the above three-dimensional structural information of trees, and tree species identification is interpreted by professional personnel visually, which has the problems of low work efficiency, large workload, easy to be affected by subjective factors, long update cycle, etc. Therefore, an automatic and intelligent method is urgently needed to replace the traditional method for urban street tree resource inventory.

[0003] Mobile measurement systems equipped with multiple sensors are widely used to collect point cloud data and street view image data of road scenes, providing important data sources for road scene understanding and high-precision three-dimensional reconstruction. With the development of deep learning, image classification based on deep learning has become mature, but the information provided by the images collected by mobile measurement systems is limited, and the complexity of road scenes is high, which increases the interference information, so the existing image classification method based on deep learning has low accuracy in automatically identifying street tree species. SUMMARY

[0004] The present application provides a multi-modal data fusion fine-grained urban tree species classification method and device to solve the problem of low accuracy in automatically identifying street tree species by existing image classification methods based on deep learning.

[0005] The first aspect of the present application provides a multi-modal data fusion fine-grained urban tree species classification method, comprising the following steps: collecting street tree original point cloud and street view image of the target street; identifying tree species information of the street tree original point cloud to construct a point cloud tree species classification dataset; performing image cropping on the street view image according to the point cloud tree species classification dataset to construct an image tree species dataset; training a pre-constructed deep learning neural network using the point cloud tree species classification dataset and the image tree species dataset until the preset requirements are met, obtaining a final tree species classification model; using the final tree species classification model to classify the tree species of the urban street tree to be classified to obtain the tree information of the urban street tree to be classified.

[0006] Optionally, the street tree original point cloud is subjected to species information identification to construct a point cloud species classification dataset, comprising:

[0007] The original point cloud of the street tree is subjected to single-tree extraction to obtain tree point clouds containing different tree species; the tree species information of the tree point clouds is identified according to the preset street tree resource inventory data, and the tree species information is assigned to the tree point clouds with a horizontal distance difference less than a preset threshold to obtain the point cloud species classification dataset.

[0008] Optionally, the street view image is subjected to image cropping according to the point cloud species classification dataset to construct an image tree species dataset, comprising:

[0009] The street view image is registered with the point cloud species classification dataset to obtain a registration relationship; based on the registration relationship, the 3D outer bounding box and the tree species information of the tree point clouds in the point cloud species classification dataset are mapped onto the street view image, and the image is cropped with the maximum range of the 3D outer bounding box on the street view image as a 2D rectangular frame to obtain the image tree species dataset.

[0010] Optionally, the point cloud species classification dataset and the image tree species dataset are used to train a pre-constructed deep learning neural network until a preset requirement is met to obtain a final tree species classification model, comprising:

[0011] The point cloud species classification dataset is divided into a first training set, a first validation set and a first test set, and the image tree species dataset is divided into a second training set, a second validation set and a second test set; the first training set and the second training set are input into the deep learning neural network to extract a plurality of point cloud features and a plurality of image features in the first training set and the second training set, respectively, and the plurality of point cloud features and the plurality of image features are connected to obtain a feature vector; the deep learning neural network is iteratively trained according to the feature vector, and the parameters of the deep learning neural network are updated according to a preset learning rate with a decay strategy until a preset requirement is met to obtain a plurality of tree species classification models; the first validation set and the second validation set are input into the plurality of tree species classification models for performance verification to select the final tree species classification model, and the first test set and the second test set are subjected to tree species classification testing on the final tree species classification model.

[0012] The second aspect embodiment of the present application provides a multi-modal data fusion fine-grained urban tree species classification device, comprising:

[0013] The collection module is configured to collect original point clouds of street trees and street view images of a target street; the identification module is configured to identify tree species information of the original point clouds of the street trees to construct a point cloud tree species classification dataset; the image cropping module is configured to crop the street view images according to the point cloud tree species classification dataset to construct an image tree species dataset; the training module is configured to train a pre-constructed deep learning neural network by using the point cloud tree species classification dataset and the image tree species dataset until a preset requirement is met, and obtain a final tree species classification model; and the classification module is configured to classify tree species of a to-be-classified urban street tree by using the final tree species classification model to obtain tree information of the to-be-classified urban street tree.

[0014] Optionally, the identification module comprises: an extraction unit configured to extract individual trees from the original point clouds of the street trees to obtain tree point clouds containing different tree species; and an identification unit configured to identify tree species information of the tree point clouds according to preset street tree resource inventory data, and assign the tree species information to tree point clouds with a horizontal distance difference less than a preset threshold to obtain the point cloud tree species classification dataset.

[0015] Optionally, the image cropping module comprises: a registration unit configured to register the street view images with the point cloud tree species classification dataset to obtain a registration relationship; and an image cropping unit configured to map 3D outer bounding boxes and tree species information of tree point clouds in the point cloud tree species classification dataset to the street view images based on the registration relationship, and crop the street view images as 2D rectangular boxes with a maximum range of the 3D outer bounding boxes on the street view images to obtain the image tree species dataset.

[0016] Optionally, the training module comprises: a division unit configured to divide the point cloud tree species classification dataset into a first training set, a first validation set and a first test set, and divide the image tree species dataset into a second training set, a second validation set and a second test set; a feature extraction unit configured to input the first training set and the second training set into the deep learning neural network to extract a plurality of point cloud features and a plurality of image features in the first training set and the second training set respectively, and connect the plurality of point cloud features and the plurality of image features to obtain a feature vector; a training unit configured to iteratively train the deep learning neural network according to the feature vector, and update parameters of the deep learning neural network according to a preset learning rate with a decay strategy until a preset requirement is met, and obtain a plurality of tree species classification models; and a verification and test unit configured to input the first validation set and the second validation set into the plurality of tree species classification models for performance verification to select the final tree species classification model, and perform tree species classification test on the final tree species classification model by using the first test set and the second test set.

[0017] The third aspect of the present application provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for fine-grained urban tree species classification based on multi-modal data fusion as described in the above embodiments.

[0018] The fourth aspect of the present application provides a computer readable storage medium, which stores a computer program executable by a processor to implement the method for fine-grained urban tree species classification based on multi-modal data fusion as described above.

[0019] The method and device for fine-grained urban tree species classification based on multi-modal data fusion provided by the embodiments of the present application can classify urban street tree species with high precision through street tree original point cloud and street view image, and have strong robustness to incomplete tree point cloud and occlusion in images, and can quickly and accurately obtain tree species information of target trees in a severely occluded scene, and the results can be displayed in a global perspective and a local perspective, and can be applied to urban street tree resource investigation, urban above-ground biomass estimation and the like.

[0020] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0021] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, taken in conjunction with the accompanying drawings, in which:

[0022] Figure 1 A flowchart of a method for fine-grained urban tree species classification based on multi-modal data fusion provided by the embodiments of the present application;

[0023] Figure 2 A specific execution schematic diagram of a method for fine-grained urban tree species classification based on multi-modal data fusion provided by the embodiments of the present application;

[0024] Figure 3 A coordinate conversion schematic diagram from point cloud to image provided by the embodiments of the present application;

[0025] Figure 4 An automatic removal range schematic diagram of a cropped image provided by the embodiments of the present application;

[0026] Figure 5 A structure schematic diagram of a pre-constructed deep learning neural network provided by the embodiments of the present application;

[0027] Figure 6A block schematic diagram of a multi-modal data fusion fine-grained urban tree species classification device provided by an embodiment of the present application is shown in the figure.

[0028] Figure 7 A structural schematic diagram of an electronic device provided by an embodiment of the present application is shown in the figure.

[0029] Legend of reference signs:

[0030] 60 - multi-modal data fusion fine-grained urban tree species classification device, 601 - acquisition module, 602 - recognition module, 603 - image cropping module, 604 - training module, and 605 - classification module. DETAILED DESCRIPTION

[0031] Embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, in which the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by reference to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as limiting the present application.

[0032] A multi-modal data fusion fine-grained urban tree species classification method and device of an embodiment of the present application are described below with reference to the accompanying drawings.

[0033] Figure 1 A flowchart of a multi-modal data fusion fine-grained urban tree species classification method provided by an embodiment of the present application is shown in the figure.

[0034] As shown in the figure, the multi-modal data fusion fine-grained urban tree species classification method includes the following steps: Figure 1

[0035] In step S101, the original point cloud of street trees and the street view image of the target street are acquired.

[0036] In step S102, tree species information recognition is performed on the original point cloud of street trees to construct a point cloud tree species classification data set.

[0037] In some embodiments, tree species information recognition is performed on the original point cloud of street trees to construct a point cloud tree species classification data set, including:

[0038] In some embodiments, tree species information recognition is performed on the original point cloud of street trees to construct a point cloud tree species classification data set, including:

[0039] According to the preset street tree resource inventory data, the tree species information of the tree point cloud is identified, and the tree species information is assigned to the tree point cloud with a horizontal distance difference less than a preset threshold, to obtain a point cloud tree species classification data set.

[0040] As shown in the figure, the multi-modal data fusion fine-grained urban tree species classification method includes the following steps: Figure 2 ​As shown, in actual execution, the original point cloud of street trees and the street view image of the target street can be scanned by the vehicle-mounted laser radar, the original point cloud of street trees is divided into blocks according to the moving track of the vehicle-mounted laser radar during data collection at a preset interval (for example, 100 meters), the point cloud after division is obtained, then a single tree extraction algorithm based on confidence estimation is used to extract single trees from the point cloud after division, to obtain tree point clouds containing different tree species, and the position information of the trees predicted by the algorithm is saved.

[0041] The extracted tree point clouds are manually cleaned to eliminate point clouds with extraction errors such as missing and redundancy, and the tree species information is assigned to the tree point clouds with a horizontal distance difference less than 0.1 m according to the street tree resource inventory data (including tree position, tree species, etc.) or combined with manual visual identification of point clouds and images, to form a point cloud tree species classification data set.

[0042] In step S103, the street view image is image cropped according to the point cloud tree species classification data set to construct an image tree species data set.

[0043] In some embodiments, the street view image is image cropped according to the point cloud tree species classification data set to construct an image tree species data set, including:

[0044] The street view image is registered with the point cloud tree species classification data set to obtain a registration relationship;

[0045] Based on the registration relationship, the 3D outer bounding box of the tree point cloud and the tree species information in the point cloud tree species classification data set are mapped to the street view image, and the maximum range of the 3D outer bounding box on the street view image is taken as a 2D rectangular frame for image cropping to obtain the image tree species data set.

[0046] Specifically, the two-dimensional projection coordinates of the three-dimensional point cloud on the image are obtained according to the conversion relationship from the world coordinate system to the camera coordinate system and from the camera coordinate system to the image coordinate system, to register the street view image with the point cloud tree species classification data set, and the registration process is as follows:

[0047] The point cloud data is converted from the world coordinate system to the sensor coordinate system, and the conversion formula is as follows:

[0048]

[0049] In the formula, X c , Y c , and Z c are the coordinates in the sensor coordinate system, X w , Y w , and Z w are the coordinates in the world coordinate system, R and T are the rotation matrix and translation vector respectively, α, β, and γ are the included angles between the x, y, and z axes of the two coordinate systems respectively, T x , and Ty 、T z It is the coordinate of the exposure position of the photo in the world coordinate system.

[0050] Furthermore, the conversion from the sensor coordinate system to the image coordinate system is realized. Figure 3 As shown, the image coordinate system is established with the upper left corner of the image as the coordinate origin, the horizontal side as the u-axis, and the vertical side as the v-axis. The conversion formula is as follows:

[0051]

[0052] Where φ and θ represent the line connecting a point A in space and the center of the camera coordinate system and the coordinate axis Z, respectively. c The angle between the projection of the line on the surface and the coordinate axis X c imagewidth and imageheight represent the width and height of the corresponding panoramic image.

[0053] The two-step transformation relationship described above can be used to obtain the two-dimensional projection coordinates of the three-dimensional point cloud on the image, thereby realizing the registration of the point cloud and street view imagery.

[0054] like Figure 2 As shown in the figure, based on the point cloud tree species dataset, the 3D minimum bounding box of the tree point cloud is mapped onto a 2D image, and its maximum bounding rectangle on the image is obtained. The street view image is cropped based on the rectangular box, and undersized cropped images are automatically removed and / or heavily occluded cropped images are manually removed to form the image tree species dataset. Finally, the tree species information from the point cloud is assigned to the cropped images, completing the creation of the image tree species dataset.

[0055] In the implementation, the translation and rotation matrices required to convert the world coordinate system to the camera coordinate system are derived based on the pitch, roll, yaw, and exposure position of the image exposure file. Cropped images that are too small are automatically removed. Each street view image is sized at 8192×4096 pixels, and cropped images of trees falling within 1024 pixels on either side and 2048 pixels in the center are discarded.

[0056] In step S104, the pre-built deep learning neural network is trained using the point cloud tree species classification dataset and the image tree species dataset until the preset requirements are met to obtain the final tree species classification model.

[0057] In some embodiments, a pre-built deep learning neural network is trained using a point cloud tree species classification dataset and an image tree species dataset until preset requirements are met to obtain a final tree species classification model, including:

[0058] The point cloud tree species classification dataset is divided into a first training set, a first validation set and a first test set, and the image tree species dataset is divided into a second training set, a second validation set and a second test set;

[0059] The first training set and the second training set are input into the deep learning neural network to extract a plurality of point cloud features and a plurality of image features in the first training set and the second training set respectively, and the plurality of point cloud features and the plurality of image features are connected to obtain a feature vector;

[0060] The deep learning neural network is iteratively trained according to the feature vector, and the parameters of the deep learning neural network are updated according to a preset learning rate with a decay strategy until a preset requirement is met, to obtain a plurality of tree species classification models;

[0061] The first validation set and the second validation set are input into the plurality of tree species classification models for performance verification to select a final tree species classification model, and the first test set and the second test set are used for tree species classification testing of the final tree species classification model.

[0062] In step S105, the final tree species classification model is used to classify the urban street trees to be classified to obtain tree information of the urban street trees to be classified.

[0063] Specifically, the obtained point cloud tree species classification dataset can be divided into a first training set, a first test set and a first validation set according to a ratio of 8:1:1, and the image tree species dataset is divided according to its corresponding point cloud to obtain a second training set, a second test set and a second validation set. Finally, the first training set and the second training set are used to train a pre-constructed deep learning neural network that fuses point clouds and images.

[0064] As shown in Figure 4 and Figure 5 Residual neural network ResNet and point cloud deep hierarchical feature learning network PointNet++ in the deep learning neural network are used as feature extractors of the first training set and the second training set respectively, that is, 1x2048-dimensional features of image data are extracted by ResNet, and 1x1024-dimensional features of point cloud data are extracted by PointNet++. Then, the plurality of point cloud features and the plurality of image features are connected to obtain a 1x3072-dimensional feature vector. Finally, the 1x3072-dimensional feature vector is input into a multi-layer perceptron network MLP for iterative training to obtain a prediction probability of each tree species. The parameters of the deep learning neural network are updated according to a preset learning rate with a decay strategy until a preset requirement is met, to obtain a plurality of tree species classification models.

[0065] The first verification set and the second verification set are input into the plurality of tree species classification models for performance verification, so as to select a final tree species classification model, and the first test set and the second test set are used for tree species classification test on the final tree species classification model. Finally, the final tree species classification model can be used for tree species classification on the trees to be classified, so as to obtain tree information of the trees to be classified.

[0066] In the specific implementation process, in the model setting in the training process, the initial learning rate is set to 0.001, and the learning rate is decayed by 0.9 times with a step of 5 training periods, and the decay is not performed when the learning rate is less than 0.0001. The optimization function is selected as Adam, which can adaptively obtain a learning rate for each parameter in the model training and optimization process, so as to realize the double improvement of optimization quality and speed. Considering the imbalance of the sample categories of the training data set, the Focal Loss is used as the loss function, and a total of 100 cycles (i.e. the preset requirement) are trained, and finally the verification set is used to select the final tree species classification model from the plurality of tree species classification models.

[0067] In the data input aspect, appropriate point cloud enhancement processing is performed when the point cloud data is input for training and verification, including point cloud random discarding, random translation and random scaling. When the image data in the image tree species data set is input, the standardization processing is performed according to the mean and variance of all images in the second training set.

[0068] In the final tree species classification model test aspect, since the tree species classification is performed on a single tree, the precision evaluation reference value is small when a single image is used as a unit, and in most cases, a single tree corresponds to multiple image data, so the classification precision is evaluated in units of single trees. Specifically, for a single tree, the prediction information label of each image or each image point cloud combination is obtained, the label is counted, and the label with the largest number is taken as the final prediction information label of the tree by the final tree species classification model. If the prediction label with the largest number is not unique, the cumulative value of the prediction probability is compared, and the label corresponding to the highest value is taken as the final prediction information.

[0069] In summary, according to the fine-grained urban tree species classification method based on multi-modal data fusion provided in the embodiments of the present application, the original point cloud of the street tree and the street view image are used to perform high-precision classification on the tree species of the street tree in the city, which has strong robustness to incomplete tree point cloud and image occlusion, and can quickly and accurately obtain the tree species information of the target tree in a severely occluded scene. The results can be displayed in a global view and a local view, and can be applied to urban street tree resource investigation, urban above-ground biomass estimation and the like.

[0070] Secondly, the fine-grained urban tree species classification device based on multi-modal data fusion provided in the embodiments of the present application is described with reference to the accompanying drawings.

[0071] Figure 6 is a block schematic diagram of a multi-modal data fusion fine-grained urban tree species classification device according to an embodiment of the present application.

[0072] As shown in Figure 6 The multi-modal data fusion fine-grained urban tree species classification device 60 comprises a collection module 601, an identification module 602, an image cropping module 603, a training module 604 and a classification module 605.

[0073] The collection module 601 is configured to collect street tree original point clouds and street view images of a target street. The identification module 602 is configured to identify tree species information of the street tree original point clouds to construct a point cloud tree species classification dataset. The image cropping module 603 is configured to crop the street view images according to the point cloud tree species classification dataset to construct an image tree species dataset. The training module 604 is configured to train a pre-constructed deep learning neural network using the point cloud tree species classification dataset and the image tree species dataset until a preset requirement is met to obtain a final tree species classification model. The classification module 605 is configured to classify tree species of a to-be-classified urban street tree using the final tree species classification model to obtain tree information of the to-be-classified urban street tree.

[0074] In some embodiments, the identification module 602 comprises:

[0075] An extraction unit configured to extract individual trees from the street tree original point clouds to obtain tree point clouds containing different tree species;

[0076] An identification unit configured to identify tree species information of the tree point clouds according to preset street tree resource inventory data, and assign the tree species information to the tree point clouds with a horizontal distance difference less than a preset threshold to obtain the point cloud tree species classification dataset.

[0077] In some embodiments, the image cropping module 603 comprises:

[0078] A registration unit configured to register the street view images with the point cloud tree species classification dataset to obtain a registration relationship;

[0079] An image cropping unit configured to map 3D bounding boxes and tree species information of the tree point clouds in the point cloud tree species classification dataset onto the street view images based on the registration relationship, and crop the images based on the maximum range of the 3D bounding boxes on the street view images as 2D rectangular boxes to obtain the image tree species dataset.

[0080] In some embodiments, the training module 604 comprises:

[0081] A division unit configured to divide the point cloud tree species classification dataset into a first training set, a first validation set and a first test set, and divide the image tree species dataset into a second training set, a second validation set and a second test set;

[0082] a feature extraction unit configured to input the first training set and the second training set into the deep learning neural network to extract a plurality of point cloud features and a plurality of image features in the first training set and the second training set respectively, and connect the plurality of point cloud features and the plurality of image features to obtain a feature vector;

[0083] a training unit configured to iteratively train the deep learning neural network according to the feature vector, and update parameters of the deep learning neural network according to a preset learning rate with a decay strategy until a preset requirement is met, to obtain a plurality of tree species classification models;

[0084] a verification and test unit configured to input the first verification set and the second verification set into the plurality of tree species classification models to perform performance verification, to select a final tree species classification model, and perform tree species classification test on the final tree species classification model by using the first test set and the second test set.

[0085] It should be noted that the above description of the embodiment of the multi-modal data fusion fine-grained urban tree species classification method is also applicable to the multi-modal data fusion fine-grained urban tree species classification device of the embodiment, which will not be described here.

[0086] The multi-modal data fusion fine-grained urban tree species classification device provided by the embodiment of the present application can classify urban street tree species with high precision through street tree original point cloud and street view image, has strong robustness to incomplete tree point cloud and occlusion in image, and can quickly and accurately obtain tree species information of a target tree in a severely occluded scene, and the result can be displayed in a global view and a local view, and can be applied to urban street tree resource investigation, urban above-ground biomass estimation, etc.

[0087] Figure 7 The electronic device provided by the embodiment of the present application is shown in a structural schematic diagram. The electronic device can include:

[0088] The memory 701, the processor 702, and a computer program stored in the memory 701 and executable on the processor 702.

[0089] The processor 702 implements the multi-modal data fusion fine-grained urban tree species classification method provided in the above embodiments when executing the program.

[0090] Further, the electronic device further includes:

[0091] The communication interface 703 is configured to communicate between the memory 701 and the processor 702.

[0092] The memory 701 is configured to store the computer program executable on the processor 702.

[0093] The memory 701 can include a high-speed RAM memory, and can also include a non-volatile memory, for example, at least one disk memory.

[0094] If the memory 701, the processor 702 and the communication interface 703 are implemented independently, the communication interface 703, the memory 701 and the processor 702 can be connected to each other through a bus and complete communication between each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Figure 7 In the figure, only one thick line is used to represent, but it does not mean that there is only one bus or one type of bus.

[0095] Optionally, in a specific implementation, if the memory 701, the processor 702 and the communication interface 703 are integrated on a chip, the memory 701, the processor 702 and the communication interface 703 can complete communication between each other through an internal interface.

[0096] The processor 702 can be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present application.

[0097] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the multi-modal data fusion fine-grained urban tree species classification method as above.

[0098] In the description of the application, reference to "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" means that a particular feature, structure, material, or characteristic being described is included in at least one embodiment or example of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment or example. Furthermore, the described specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples. In addition, the usage of "first", "second" and the like does not indicate any order but rather serves merely to name various components. Moreover, the usage of "top", "bottom", and the like is made for the purpose of illustration only and the application can operate in any orientation unless otherwise stated.

[0099] Furthermore, the terms "first", "second", or the like, do not denote any quantity or order but are merely used as labels for distinguishing between the different components. Thus, these terms are used in their contextually clear and implicit meaning. The use of "N" in the description means at least two, such as two, three, etc., unless explicitly stated otherwise.

[0100] Any process or method descriptions or blocks in flow charts or otherwise described herein represent embodiments which can be managed as one or more modules, segments, or portions of code which include one or more executable instructions for implementing specific logic functions or steps, and the preferred embodiments of this application also include the combination of other steps, functions, and / or structures as can be needed to complete the described functions, and such described functions can be implemented in an order different than the specific order described herein. The described processes, methods, and / or blocks can be embodied in a machine-dependent, specific, proprietary, application- or hardware-specific, and / or operating system- specific, including a proprietary computer language, and / or by transitory signals.

[0101] The logic and / or steps represented in the flowcharts and / or described herein, for example, can be considered as a sequence of executable instructions stored in a computer readable medium, which can be executed by an instruction execution system, apparatus or device, such as a computer-based system, a processor-based system, or other system that can fetch the instructions from the instruction execution system, apparatus or device and execute the instructions, or a combination of the above. For the purposes of this specification, a "computer readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus or device. The computer readable medium can be a computer readable storage medium or a computer readable signal medium. The computer readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a propagation medium. The computer readable signal medium can include, but is not limited to, a computer readable medium that facilitates transfer of the program from one place to another. A specific example of a computer readable medium is a non-transitory computer-readable storage medium. A specific example of a computer readable signal medium is a source or destination of the computer readable medium. Another specific example of a computer readable signal medium is a computer readable signal travelling through space. Thus, a computer readable medium can take many forms of hardware to carry out the program for use by or in connection with the instruction execution system, apparatus or device.

[0102] It should be understood that aspects of the application can be implemented in hardware, software, firmware or a combination thereof. In the above embodiments, the N steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented in hardware and in another embodiment, the hardware can be implemented with any or a combination of the following technologies, which are all well known in the art: a discrete logic circuit(s) having logic gates for implementing logic functions upon an application of data signals, an application specific integrated circuit having appropriate combinational logic gates, a programmable gate array(s) (PGA), a field programmable gate array (FPGA), etc.

[0103] Those of skill in the art would understand that information and signals can be represented using any of a variety of technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that can be referenced throughout the above description can be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof.

[0104] In addition, each function unit in each embodiment of the present application can be integrated in one processing module, or each unit can be physically present separately, or two or more units can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software function module. When the integrated module is realized in the form of a software function module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0105] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A fine-grained urban tree species classification method based on multimodal data fusion, characterized by: The following steps are involved: Collect the original point cloud and street view images of the street trees on the target street; Identifying tree species information on the original point cloud of the street trees to construct a point cloud tree species classification dataset; Performing image cropping on the street view image according to the point cloud tree species classification dataset to construct an image tree species dataset; The pre-built deep learning neural network is trained using the point cloud tree species classification dataset and the image tree species dataset until preset requirements are met to obtain a final tree species classification model; The final tree species classification model is used to classify the urban roadside trees to be classified into tree species, so as to obtain tree information of the urban roadside trees to be classified.

2. The fine-grained urban tree species classification method based on multimodal data fusion according to claim 1 is characterized in that: The identifying of tree species information on the original point cloud of the street trees to construct a point cloud tree species classification dataset includes: Performing single tree extraction on the original point cloud of the street trees to obtain a tree point cloud containing different tree species; The tree species information of the tree point cloud is identified according to the preset street tree resource inventory data, and the tree species information is assigned to the tree point cloud with a horizontal distance difference less than a preset threshold to obtain the point cloud tree species classification data set.

3. The fine-grained urban tree species classification method based on multimodal data fusion according to claim 1 is characterized in that: The step of cropping the street view image according to the point cloud tree species classification dataset to construct an image tree species dataset includes: Registering the street view image with the point cloud tree species classification dataset to obtain a registration relationship; Based on the registration relationship, the 3D outer bounding box and tree species information of the tree point cloud in the point cloud tree species classification dataset are mapped to the street view image, and the maximum range of the 3D outer bounding box on the street view image is used as a 2D rectangular box for image cropping to obtain the image tree species dataset.

4. The fine-grained urban tree species classification method based on multimodal data fusion according to claim 1 is characterized in that: The method of using the point cloud tree species classification dataset and the image tree species dataset to train a pre-built deep learning neural network until preset requirements are met to obtain a final tree species classification model includes: Dividing the point cloud tree species classification dataset into a first training set, a first validation set, and a first test set, and dividing the image tree species dataset into a second training set, a second validation set, and a second test set; Inputting the first training set and the second training set into the deep learning neural network to extract a plurality of point cloud features and a plurality of image features from the first training set and the second training set, respectively, and concatenating the plurality of point cloud features with the plurality of image features to obtain a feature vector; Iteratively training the deep learning neural network according to the feature vector, and updating the parameters of the deep learning neural network according to a preset learning rate with a decay strategy until preset requirements are met, thereby obtaining multiple tree species classification models; The first validation set and the second validation set are input into the multiple tree species classification models for performance verification to select the final tree species classification model, and the first test set and the second test set are used to perform a tree species classification test on the final tree species classification model.

5. A fine-grained urban tree species classification device based on multimodal data fusion, characterized in that: include: The acquisition module is used to collect the original point cloud and street view images of the street trees on the target street; an identification module for identifying tree species information on the original point cloud of the street trees to construct a point cloud tree species classification dataset; An image cropping module, configured to crop the street view image according to the point cloud tree species classification dataset to construct an image tree species dataset; A training module is used to train a pre-built deep learning neural network using the point cloud tree species classification dataset and the image tree species dataset until preset requirements are met to obtain a final tree species classification model; The classification module is used to classify the urban roadside trees to be classified by using the final tree species classification model to obtain tree information of the urban roadside trees to be classified.

6. The fine-grained urban tree species classification device based on multimodal data fusion according to claim 5 is characterized in that: The identification module includes: An extraction unit is used to extract individual trees from the original point cloud of the roadside trees to obtain a point cloud containing trees of different species; The identification unit is used to identify the tree species information of the tree point cloud according to the preset street tree resource inventory data, and assign the tree species information to the tree point cloud with a horizontal distance difference less than a preset threshold to obtain the point cloud tree species classification data set.

7. The fine-grained urban tree species classification device based on multimodal data fusion according to claim 5 is characterized in that: The image cropping module includes: a registration unit, configured to register the street view image with the point cloud tree species classification dataset to obtain a registration relationship; An image cropping unit is configured to map the 3D outer bounding box and tree species information of the tree point cloud in the point cloud tree species classification dataset onto the street view image based on the registration relationship, and perform image cropping using the maximum range of the 3D outer bounding box on the street view image as a 2D rectangular frame to obtain the image tree species dataset.

8. The fine-grained urban tree species classification device based on multimodal data fusion according to claim 5 is characterized in that: The training module includes: a division unit, configured to divide the point cloud tree species classification dataset into a first training set, a first validation set, and a first test set, and to divide the image tree species dataset into a second training set, a second validation set, and a second test set; a feature extraction unit, configured to input the first training set and the second training set into the deep learning neural network, extract a plurality of point cloud features and a plurality of image features from the first training set and the second training set, respectively, and concatenate the plurality of point cloud features and the plurality of image features to obtain a feature vector; A training unit, configured to iteratively train the deep learning neural network according to the feature vector, and update the parameters of the deep learning neural network according to a preset learning rate with a decay strategy until preset requirements are met, thereby obtaining multiple tree species classification models; A verification and testing unit is used to input the first verification set and the second verification set into the multiple tree species classification models for performance verification to select the final tree species classification model, and use the first test set and the second test set to perform tree species classification testing on the final tree species classification model.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the fine-grained urban tree species classification method based on multimodal data fusion as described in any one of claims 1 to 4.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The computer program is executed by a processor to implement the fine-grained urban tree species classification method based on multimodal data fusion as claimed in any one of claims 1 to 4.

Images