Vegetation type recognition method based on machine vision technology and deep learning fusion

Abstract

The application discloses a vegetation type recognition method based on the fusion of machine vision technology and deep learning, which utilizes a multi-channel remote sensing spectral camera carried by a UAV to collect multispectral and hyperspectral remote sensing images of vegetation in a sample area; the multispectral and hyperspectral remote sensing images are pretreated, the pretreated images are fused by a method combining a generative adversarial network and a convolutional neural network; based on an ORB feature detection algorithm, color features, shape features and brightness features of trees, shrubs and grasses are extracted from the fused images respectively to obtain a vegetation feature image library; a single-step multi-frame target detection algorithm is used to train the vegetation feature images, and a trained vegetation type detection model is used to identify the vegetation type in a region to be identified, so that the vegetation type can be quickly identified in a field investigation process, and the vegetation type recognition method provided by the application improves the accuracy of vegetation type recognition by fusing machine vision and deep learning technology.

Description

Technical Field

[0001] This invention belongs to the field of vegetation type recognition, specifically relating to a vegetation type recognition method based on the fusion of machine vision technology and deep learning. Background Technology

[0002] Vegetation type surveys are a common problem in ecological monitoring. While traditional methods of classifying vegetation through visual observation can accurately identify vegetation types, they are extremely time-consuming, resource-intensive, and require significant manpower, material resources, and financial investment. How to quickly and accurately identify vegetation types has become a pressing issue in the field of ecological monitoring.

[0003] UAVs equipped with multispectral and hyperspectral sensors can acquire a large amount of vegetation remote sensing data. However, how to extract vegetation features after acquiring vegetation remote sensing images using UAVs is a problem that urgently needs to be solved in this field. Machine vision technology is currently an effective method for rapidly identifying vegetation types; however, due to the limitations of machine vision technology, it cannot achieve the expected accuracy. Deep learning algorithms, on the other hand, have proven to have good performance in accurately identifying vegetation types. This invention, by integrating machine vision and deep learning technologies, can accurately and quickly identify vegetation types, improving the efficiency of field vegetation type surveys. Summary of the Invention

[0004] To address the aforementioned problems in the prior art, the present invention is achieved through the following technical solution:

[0005] In a first aspect, the present invention provides a vegetation type identification method based on the fusion of machine vision technology and deep learning, comprising the following steps:

[0006] S10. Acquire images of the sample collection area, and preprocess the images through quality checks and orthorectification to form multispectral orthophotos and hyperspectral orthophotos of the sample collection area.

[0007] S20. Using a generative adversarial network and a dual-branch convolutional neural network, the multispectral orthophoto and the hyperspectral orthophoto are fused to generate a spectral image for vegetation type determination.

[0008] S30. Based on the ORB algorithm, vegetation features are extracted from the spectral image of vegetation type precision determination to generate a vegetation feature image of the spectral image of vegetation type precision determination. The vegetation features include color, shape and brightness.

[0009] S40. A vegetation type detection model is obtained by training the vegetation feature image based on the single-step multi-frame target detection algorithm.

[0010] S50. Input the spectral image of the vegetation type of the area to be identified into the vegetation type detection model, which can identify the vegetation type of the area to be identified.

[0011] S60. Set up a standard image database for sub-class vegetation. The vegetation type detection model identifies the most similar sub-class standard vegetation feature image under the vegetation type to which the vegetation in the area to be identified belongs based on the standard image database for sub-class vegetation, and obtains the most similar sub-class identification probability.

[0012] In one embodiment of the present invention, S20 includes:

[0013] S201. Input the multispectral orthophoto into the generator network in the generative adversarial network to generate a preliminary spectral image for vegetation type determination.

[0014] S202. Input the preliminary vegetation type spectral image and hyperspectral orthophoto into the feature extraction network to extract features, form the feature map of the preliminary vegetation type spectral image and the feature map of the hyperspectral image, and stitch them together to obtain the feature stitched image.

[0015] S203. Input the feature stitched image into the feature fusion network to form a feature fusion image with high target space and high preset spectral resolution;

[0016] S204. Input the feature fusion image into the feature reconstruction network to reconstruct the fused features into a feature reconstruction image with a high feature level, wherein the high feature level includes the texture detail features, semantic features and abstract features of the image;

[0017] S205. Input the feature reconstruction image into the feature reconstruction network to reconstruct the feature reconstruction image according to the preset target spatial information and target spectral information, and generate a detailed spectral image of vegetation type after feature reconstruction.

[0018] S206. Input the preliminary spectral image of vegetation type and the detailed spectral image of vegetation type into the discriminator in the generative adversarial network respectively. The discriminator distinguishes whether the preliminary spectral image of vegetation type and the detailed spectral image of vegetation type generated by the generator reach the preset prediction probability.

[0019] S207. Repeat S201 to S206 to train the generator until the preliminary vegetation type spectral image and the detailed vegetation type spectral image reach the preset prediction probability. The preliminary vegetation type spectral image forms the detailed vegetation type spectral image.

[0020] In one embodiment of the present invention, S30 includes:

[0021] S301. Use the accelerated robust feature algorithm to extract significant vegetation feature points from the spectral image of vegetation type precision determination;

[0022] S302. Assign a principal direction to each feature point to increase the invariance of the feature description of the feature point;

[0023] S303. Use a boxed filter to calculate feature points;

[0024] S304. For each feature point, the improved BRIEF descriptor is used to select feature points with large RGB value differences, brightness and contrast. Feature point orientation gradient features and local structure features are introduced to generate a binary comparison template and calculate the features of the area around the feature point. The feature points with large RGB value differences, brightness and contrast are feature points with RGB value differences higher than a preset threshold, brightness higher than a preset threshold, and contrast higher than a preset threshold.

[0025] S305. Calculate the distance and similarity between the BRIEF descriptors, and use the best candidate matching method to match the feature points in the image;

[0026] S306. Based on the matching results, the matching feature points determined in S305 are combined into feature point pairs, and a vegetation feature image with vegetation features is output.

[0027] In one embodiment of the present invention, S40 includes:

[0028] S401. Label the vegetation feature images with vegetation features generated in S306 with names to use as training data.

[0029] S402. Select the target to be identified in the vegetation feature image based on the single-step multi-frame target detection algorithm;

[0030] S403. Perform a convolution operation on each vegetation feature image to predict the category and location information of the target to be identified, and use an activation function to output the probability of each category;

[0031] S404. Train a vegetation type detection model using a non-maximum suppression algorithm. The vegetation type detection model is represented as follows:

[0032]

[0033] Where N represents the total number of samples, y i [j] represents the probability that the current input sample belongs to the j-th category, t i [j] represents the true label of the current input sample belonging to the j-th vegetation type, where j = {1, 2, 3}.

[0034] In one embodiment of the present invention, the activation function is the softmax activation function.

[0035] In a second aspect of the invention, an electronic device is provided, including a processor and a memory, the memory for storing one or more programs, the one or more programs including step instructions according to the vegetation type identification method based on the fusion of machine vision technology and deep learning, the processor for executing the one or more programs.

[0036] In a third aspect of the invention, a storage medium is provided storing a computer program that, when executed by a processor, enables the implementation of the steps of the vegetation type identification method based on the fusion of machine vision technology and deep learning.

[0037] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0038] This invention provides a vegetation type identification method based on the fusion of machine vision technology and deep learning. It utilizes a UAV equipped with a multi-channel remote sensing spectral camera to acquire multispectral and hyperspectral remote sensing images of vegetation in a sample area. The multispectral and hyperspectral images are then preprocessed to improve accuracy. Next, a combined generative adversarial network (GAN) and convolutional neural network (CNN) approach is used to fuse the preprocessed images. Based on OpenCV's comprehensive and universal machine learning library and the efficient ORB feature detection algorithm, color, brightness, and shape features of trees, shrubs, and grasses are extracted from the fused images, outputting vegetation feature images. The SURF local description feature algorithm is used to improve computational efficiency while ensuring stable and reliable feature recognition. A single-step multi-box object detection algorithm is used to train an object detection model on the vegetation feature images. The trained object detection model is then used to distinguish vegetation types in the area to be identified, enabling rapid identification of vegetation types in the field. The progressive processing of machine vision technology and deep learning gradually improves the accuracy of vegetation identification.

[0039] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating the steps of a vegetation type identification method based on the fusion of machine vision technology and deep learning, as provided in an embodiment of the present invention.

[0041] Figure 2 This is a schematic diagram of the image preprocessing process in the vegetation type recognition method based on the fusion of machine vision technology and deep learning provided in the embodiments of the present invention.

[0042] Figure 3This is a schematic diagram of the multispectral hyperspectral image fusion processing in the vegetation type identification method based on the fusion of machine vision technology and deep learning provided in the embodiments of the present invention. Detailed Implementation

[0043] To further illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the following detailed description of the solution according to the present invention is provided in conjunction with the accompanying drawings and specific embodiments.

[0044] The foregoing and other technical contents, features, and effects of the present invention will be clearly presented in the following detailed description of specific embodiments in conjunction with the accompanying drawings. Through the description of the specific embodiments, a more in-depth and concrete understanding can be gained of the technical means and effects adopted by the present invention to achieve its intended purpose. However, the accompanying drawings are for reference and illustration only and are not intended to limit the technical solutions of the present invention.

[0045] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations are intended to cover non-exclusive inclusion, such that an article or apparatus comprising a list of elements includes not only those elements but also other elements not expressly listed. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the article or apparatus that includes said element.

[0046] Example 1

[0047] Please see Figure 1 , Figure 1 This is a flowchart illustrating the steps of a vegetation type identification method based on the fusion of machine vision technology and deep learning, as provided in an embodiment of the present invention. The vegetation identification method includes the following steps:

[0048] S10. Acquire images of the sample collection area, and preprocess the images through quality checks and orthorectification to form multispectral orthophotos and hyperspectral orthophotos of the sample collection area.

[0049] Specifically, using a drone equipped with a multi-channel remote sensing spectral camera to acquire multispectral and hyperspectral remote sensing images of the sample collection area allows for the acquisition of rich image data while saving costs and resources. The acquired multispectral and hyperspectral images undergo preprocessing; specifically, multi-channel refers to multiple spectral bands, such as... Figure 2As shown, the acquired multispectral and hyperspectral remote sensing images are preprocessed through quality inspection and orthorectification. This preprocessing includes a series of operations such as quality inspection, aerial triangulation, aerial triangulation refinement calculation, and orthorectification. The specific steps are as follows:

[0050] S101. Quality inspection includes performing quality checks on the acquired hyperspectral and multispectral remote sensing images, and removing useless images. In this embodiment, the UAV image processing software Pix4Dmapper is used for quality inspection.

[0051] S102. Aerial triangulation includes selecting a small number of field control points from the hyperspectral and multispectral images with regional overlap after quality inspection, to obtain the height and planar orientation of the densified points in the hyperspectral and multispectral images after quality inspection.

[0052] S103, Aerial Triangulation Encryption Calculation and Orthophoto Projection, includes creating an aerial triangulation project, performing aerial triangulation encryption calculations on the hyperspectral and multispectral images after quality inspection to generate dense point clouds of the sample collection area, classifying the dense point clouds to form high-quality multispectral orthophotos and hyperspectral orthophotos of the sample collection area, thereby reducing the problems of mottled and distorted vegetation.

[0053] S20. Using generative adversarial networks and bi-branch convolutional neural networks, multispectral orthophotos and hyperspectral orthophotos are fused to generate spectral images for precise vegetation type determination.

[0054] Specifically, this generative adversarial network includes a generator and a discriminator. The generator mainly comprises a feature extraction module, a nonlinear feature extraction module, and a reconstruction module connected in sequence. Hyperspectral orthophotos and multispectral orthophotos are fused together, such as... Figure 3 As shown, the specific steps include:

[0055] S201. Input the multispectral orthophoto into the generator network in the generative adversarial network to generate a preliminary spectral image for vegetation type determination.

[0056] S202. Input the preliminary vegetation type spectral image and hyperspectral orthophoto into the feature extraction network to extract features, form the feature map of the preliminary vegetation type spectral image and the feature map of the hyperspectral image, and stitch them together to obtain the feature stitched image.

[0057] Specifically, the feature extraction network includes two branches, each consisting of three convolutional layers and one downsampling layer. The preliminary spectral image of vegetation type and the hyperspectral orthophoto are input into the generator in the generative adversarial network to extract image features. Specifically, low-dimensional features are extracted first, such as the color, brightness, and shape of vegetation. The preliminary spectral image of vegetation type and the hyperspectral orthophoto are input into a convolutional layer with a kernel size of 3×3 and a stride of 1 to extract low-dimensional features. Subsequently, the preliminary spectral image of vegetation type and the hyperspectral orthophoto are input into a nonlinear feature extraction module for convolution operation. In this nonlinear feature extraction module, the convolutional kernel size is 3×3 and the stride is 1 to extract deep features. These deep features are used to characterize the abstract features in the preliminary spectral image of vegetation type and the hyperspectral orthophoto, such as spatial information and semantic features in the image information. The feature reconstruction module includes a multi-scale residual block, an upsampling layer, and two convolutional layers. First, the residual block is input to enhance the correlation between image space and dimension. Then, the image is input to the upsampling layer to increase the image size. Finally, the image is obtained by passing through two convolutional layers to obtain the feature-stitched image.

[0058] S203. Input the feature stitched image into the feature fusion network to form a feature fusion image with high target space and high preset spectral resolution;

[0059] Specifically, the high target space refers to a spatial resolution of 1mm to 10cm, and the high preset spectral resolution refers to a resolution of 30 million to 50 million pixels. The feature fusion network includes four convolutional layers, one downsampling layer, and one upsampling layer. It fuses the spatial and spectral information of the initial spectral image of vegetation type to generate a feature fusion image, thereby enabling the feature fusion image to have high target space and high preset spectral resolution.

[0060] S204. Input the feature fusion image into the feature reconstruction network to reconstruct the fused features into a feature reconstruction image with a high feature level, wherein the high feature level includes the texture detail features, semantic features and abstract features of the image;

[0061] Specifically, the fused image is input into a feature reconstruction network, which consists of three convolutional layers and one upsampling layer. This network reconstructs the high-level feature image from the fused features. The upsampling layer is used to prevent the loss of high-frequency information. Further, residual blocks are first input to enhance the spatial and dimensional correlation of the image, then input into the upsampling layer to increase the image size, and finally passed through two convolutional layers to obtain a high-level feature reconstruction image. This embodiment uses an upsampling layer to prevent the loss of high-frequency information and reconstructs the high-level feature image from the fused features in the fused image.

[0062] S205. Input the feature reconstruction image into the feature reconstruction network to reconstruct the feature reconstruction image according to the preset target spatial information and target spectral information, and generate a detailed spectral image of vegetation type after feature reconstruction.

[0063] In this embodiment, the feature reconstruction network uses 5×5 convolutional kernels to reconstruct the feature reconstruction image. Specifically, the spatial and spectral information of the feature reconstruction image is reconstructed to obtain a high-resolution spectral image of vegetation type detailed judgment.

[0064] S206. Input the preliminary spectral image of vegetation type and the detailed spectral image of vegetation type into the discriminator in the generative adversarial network respectively. The discriminator distinguishes whether the preliminary spectral image of vegetation type and the detailed spectral image of vegetation type generated by the generator reach the preset prediction probability.

[0065] In this embodiment, the discriminator first uses a convolutional layer with a kernel size of 3 and a stride of 1 to extract features from the input image; then it uses a convolutional layer with a kernel size of 4 and a stride of 2 to reduce the spatial size of the feature map; then it uses an upsampling layer and a convolutional layer to gradually expand the size of the feature map; finally, it uses two convolutional layers with a kernel size of 3 and a stride of 1 to extract more information.

[0066] S207. Repeat S201 to S206 to train the generator until the preliminary vegetation type spectral image and the detailed vegetation type spectral image reach the preset prediction probability. The preliminary vegetation type spectral image forms a detailed vegetation type spectral image. The trained generator is used to generate hyperspectral data. The detailed vegetation type spectral image generated by the generator is used as the fusion processing result of the multispectral orthophoto and the hyperspectral orthophoto, thereby realizing the fusion of hyperspectral and multispectral images of the sample collection area.

[0067] The vegetation identification method provided in this embodiment uses deep learning fusion. Specifically, it uses generative adversarial networks and bi-branch convolutional neural networks to fuse and train preprocessed multispectral orthophotos and hyperspectral orthophotos to form a vegetation type precision spectral image. This vegetation type precision spectral image is used as the fused image of the sample collection area. This vegetation type precision spectral image combines the advantages of multispectral and hyperspectral images of the sample collection area, producing a clearer and more informative image, improving the readability and understandability of the sample collection area image, and improving the accuracy of subsequent vegetation identification.

[0068] S30. Based on the ORB algorithm, vegetation features are extracted from the spectral image of vegetation type precision determination to generate a vegetation feature image of the spectral image of vegetation type precision determination. The vegetation features include color, shape and brightness.

[0069] Vegetation feature extraction from spectral images of vegetation type precision determination based on the ORB algorithm includes the following steps:

[0070] S301. Use an accelerated robust feature algorithm to extract significant vegetation feature points from the spectral image of vegetation type precision determination;

[0071] The Speeded Up Robust Features (SURF) algorithm is a local descriptive feature algorithm that can process feature recognition from multiple images and offers advantages such as fast recognition speed, good robustness, and scale invariance. In this embodiment, the SURF algorithm includes scale-space extremum detection, uses the Hessian matrix to calculate the Laplacian transform in the image, finds candidate feature points, and locates these points. By employing the SURF algorithm, computational efficiency is improved while ensuring stable and reliable feature recognition.

[0072] S302. Assign a principal direction to each feature point to increase the invariance of the feature description of the feature point;

[0073] S303. Use a boxed filter to calculate feature points;

[0074] In this embodiment, a boxed filter is used to accelerate feature description calculation, thereby improving the efficiency of the algorithm.

[0075] S304. For each feature point, use the improved BRIEF descriptor to select feature points with large RGB value differences, brightness, and contrast. Introduce feature point orientation gradient features and local structure features to generate a binary comparison template and calculate the features of the region surrounding the feature point. The feature point with large RGB value differences, brightness, and contrast is a feature point whose RGB value difference is higher than a preset threshold, whose brightness is higher than a preset threshold, and whose contrast is higher than a preset threshold.

[0076] In this embodiment, for each feature point, the improved BRIEF descriptor selects feature points with large RGB value differences, brightness, and contrast. This allows the BRIEF descriptor to adopt an improved feature point strategy, selecting feature points with more information, thereby improving the BRIEF descriptor's discriminative power and accuracy for feature points. At the same time, the introduction of feature point orientation gradient features and local structural features enhances the BRIEF descriptor's discriminative ability, thereby improving the accuracy of vegetation type identification.

[0077] S305. Calculate the distance and similarity between the BRIEF descriptors, and use the best candidate matching method to match the feature points in the image;

[0078] S306. Based on the matching results, the matching feature points determined in step S305 are combined into feature point pairs, and a vegetation feature image with vegetation features is output.

[0079] In this embodiment, after using the downloaded Python environment and OpenCV library, the OpenCV library is imported into the Python environment. Then, the processed vegetation feature image is read, ORB feature detection is performed, and the color, shape, and brightness features of trees, shrubs, and grasses are extracted from the image based on OpenCV machine vision technology. Finally, a vegetation feature image with vegetation features is output.

[0080] S40. A vegetation type detection model is obtained by training the vegetation feature image based on the single-step multi-frame target detection algorithm.

[0081] Based on the Single Shot Multi-Box Detector (SSD) algorithm, the target type is determined. A vegetation type detection model is used to train vegetation feature images to obtain the vegetation type to which the feature images belong. This vegetation type includes trees, shrubs, and grasses. Specifically, the steps include:

[0082] S401. Label the vegetation feature images with vegetation features generated in step S306 with names to use as training data.

[0083] S402. Select the target to be identified in the vegetation feature image based on the single-step multi-frame target detection algorithm;

[0084] S403. Perform a convolution operation on each vegetation feature image to predict the category and location information of the target to be identified, and use an activation function to output the probability of each category;

[0085] In this embodiment, the activation function is the softmax activation function.

[0086] S404. Train the vegetation type detection model using the Non-Maximum Suppression (NMS) algorithm. The vegetation type detection model is represented as follows:

[0087]

[0088] Specifically, N represents the total number of samples, y i [j] represents the probability that the current input sample belongs to the j-th category, t i[j] represents the true label of the current input sample belonging to the j-th category, where j = {1, 2, 3}. The value of j is used to characterize the type of vegetation: tree, shrub, or grass. In this embodiment, a true label value of 1 represents a positive class value, and a true label value of 0 represents a negative class value. The first Σ represents the summation of all samples, and the second Σ represents the summation of all categories. The vegetation recognition method provided in this embodiment is implemented using machine vision technology. Specifically, based on the SSD algorithm, the target type is determined, and a vegetation type detection model is used to train the vegetation feature image to obtain the vegetation type to which the vegetation feature image belongs. Based on machine vision technology, a safe, reliable, and high-precision vegetation classification is achieved.

[0089] S50. After the model is trained, the spectral image of the vegetation type of the area to be identified is input into the vegetation type detection model. The vegetation type detection model can identify the vegetation type of the area to be identified.

[0090] Specifically, after the model is trained, the input is a spectral image of the vegetation type of the area to be identified, which enables rapid identification of vegetation type in the field.

[0091] S60. Set up a standard image database for subclass vegetation. The vegetation type detection model identifies the most similar subclass standard vegetation feature image under the vegetation type to which the vegetation in the area to be identified belongs based on the standard image database for subclass vegetation, and obtains the most similar subclass identification probability.

[0092] In this embodiment, vegetation feature images are implemented using the SSD algorithm based on a Python environment and the OpenCV library. The specific steps include: First, the vegetation feature images are labeled with names using the labeling tool "labellmg" and saved as XML files as training data. After all labeling is completed, the dataset is divided into training and test sets. Next, SSD is configured. Specifically, the SSD-Tensorflow folder is downloaded, the voc_2007 format files are converted to tfrecord format, the label names of the vegetation in S401 are added to the SSD-Tensorflow-master folder, SSD parameters are assigned, and all other irrelevant categories in VOC_LABELS are deleted. Then, classification and category prediction are performed. Specifically, prediction is performed using convolutional and fully connected layers. For each anchor box, the possible vegetation categories are predicted, and the probability of each category is output using the softmax activation function. Finally, the model is trained based on the SSD algorithm. Specifically, after preparing the label files and the SSD model, the prediction results are filtered and merged using a non-maximum suppression algorithm based on the classification and regression prediction results. A sub-category vegetation standard image database is established, specifically for trees, shrubs, and grasses. A vegetation type detection model is used to identify the sub-category standard vegetation feature image most similar to the vegetation in the area to be identified. In this embodiment, the Chinese Academy of Sciences Plant Image Database is used as the sub-category vegetation standard image database for trees, shrubs, and grasses. The link to the Chinese Academy of Sciences Plant Image Database is set to https: / / ppbc.iplant.cn / special. The images of the vegetation in the area to be identified are subjected to feature extraction according to the above steps, and then compared and matched with the vegetation in the sub-category vegetation standard image database to determine which sub-category standard image is most similar to the image of the vegetation in the area to be identified, thus providing the identification result.

[0093] This embodiment presents a vegetation type identification method based on the fusion of machine vision technology and deep learning. It utilizes a UAV equipped with a multi-channel remote sensing spectral camera to collect multispectral and hyperspectral remote sensing images of vegetation in a sample area, providing rich and high-quality sample support as the basic data source. The multispectral and hyperspectral remote sensing images are then preprocessed to improve the accuracy of the acquired images. Next, a combined generative adversarial network (GAN) and convolutional neural network (CNN) method is used to fuse the preprocessed images. Based on OpenCV's comprehensive and universal machine learning library and the efficient ORB feature detection algorithm, the color, brightness, and shape features of trees, shrubs, and grasses are extracted from the fused images, and vegetation feature images with vegetation characteristics are output. The SURF local description feature algorithm is used to improve computational efficiency while ensuring stable and reliable feature recognition. A single-step multi-box object detection algorithm is used to train an object detection model on the vegetation feature images. The trained object detection model is then used to distinguish the vegetation type in the area to be identified, achieving rapid identification of vegetation types in the field. By combining machine vision technology and deep learning in a progressive processing step, the accuracy of vegetation identification is gradually improved.

[0094] Example 2

[0095] This embodiment provides an electronic device based on Embodiment 1. The electronic device includes a processor and a memory. The memory is used to store one or more programs. The one or more programs include step instructions according to any of the vegetation type identification methods based on the fusion of machine vision technology and deep learning in Embodiment 1. The processor is used to execute the one or more programs.

[0096] Example 3

[0097] This embodiment, based on Embodiment 1, provides a storage medium storing a computer program that, when executed by a processor, can implement the steps of any of the vegetation type identification methods based on the fusion of machine vision technology and deep learning in Embodiment 1.

[0098] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A vegetation type identification method based on the fusion of machine vision technology and deep learning, characterized in that, include: S10. Acquire images of the sample collection area, and preprocess the images through quality checks and orthorectification to form multispectral orthophotos and hyperspectral orthophotos of the sample collection area. S20. Using a generative adversarial network and a dual-branch convolutional neural network, the multispectral orthophoto and the hyperspectral orthophoto are fused to generate a spectral image for vegetation type determination. S30. Based on the ORB algorithm, extract vegetation features from the spectral image of the vegetation type to generate a vegetation feature image of the spectral image of the vegetation type, wherein the vegetation features include color, shape, and brightness; the specific steps are as follows: S301. Use an accelerated and robust feature algorithm to extract significant vegetation feature points from the spectral image of vegetation type precision determination; S302. Assign a principal direction to each feature point to increase the invariance of the feature description of the feature point; S303. Use a boxed filter to calculate feature points; S304. For each feature point, the improved BRIEF descriptor is used to select feature points with large RGB value differences, brightness and contrast. Feature point orientation gradient features and local structure features are introduced to generate a binary comparison template and calculate the features of the area around the feature point. The feature points with large RGB value differences, brightness and contrast are feature points with RGB value differences higher than a preset threshold, brightness higher than a preset threshold, and contrast higher than a preset threshold. S305. Calculate the distance and similarity between the BRIEF descriptors, and use the best candidate matching method to match the feature points in the image; S306. Based on the matching results, the matching feature points determined in S305 are combined into feature point pairs, and a vegetation feature image with vegetation features is output. S40. A vegetation type detection model is obtained by training the vegetation feature image based on the single-step multi-frame target detection algorithm, wherein the vegetation type includes trees, shrubs and grasses; S50. Input the spectral image of the vegetation type of the area to be identified into the vegetation type detection model, which can identify the vegetation type of the area to be identified. S60. Set up a standard image database for sub-class vegetation. The vegetation type detection model identifies the most similar sub-class standard vegetation feature image under the vegetation type to which the vegetation in the area to be identified belongs based on the standard image database for sub-class vegetation, and obtains the most similar sub-class identification probability.

2. The vegetation type identification method based on the fusion of machine vision technology and deep learning according to claim 1, characterized in that, S20 includes: S201. Input the multispectral orthophoto into the generator network in the generative adversarial network to generate a preliminary spectral image for vegetation type determination. S202. Input the preliminary vegetation type spectral image and hyperspectral orthophoto into the feature extraction network to extract features, form the feature map of the preliminary vegetation type spectral image and the feature map of the hyperspectral image, and stitch them together to obtain the feature stitched image. S203. Input the feature stitched image into the feature fusion network to form a feature fusion image with high target space and high preset spectral resolution; S204. Input the feature fusion image into the feature reconstruction network to reconstruct the fused features into a feature reconstruction image with a high feature level, wherein the high feature level includes the texture detail features, semantic features and abstract features of the image; S205. Input the feature reconstruction image into the feature reconstruction network to reconstruct the feature reconstruction image according to the preset target spatial information and target spectral information, and generate a detailed spectral image of vegetation type after feature reconstruction. S206. Input the preliminary spectral image of vegetation type and the detailed spectral image of vegetation type into the discriminator in the generative adversarial network respectively. The discriminator distinguishes whether the preliminary spectral image of vegetation type and the detailed spectral image of vegetation type generated by the generator reach the preset prediction probability. S207. Repeat S201~S206 to train the generator until the initial vegetation type spectral image and the detailed vegetation type spectral image reach the preset prediction probability. The initial vegetation type spectral image forms the detailed vegetation type spectral image.

3. The vegetation type identification method based on the fusion of machine vision technology and deep learning according to claim 1, characterized in that, S40 includes: S401. Label the vegetation feature images with vegetation features generated in S306 with names to use as training data. S402. Select the target to be identified in the vegetation feature image based on the single-step multi-frame target detection algorithm; S403. Perform a convolution operation on each vegetation feature image to predict the category and location information of the target to be identified, and use an activation function to output the probability of each category; S404. Train a vegetation type detection model using a non-maximum suppression algorithm. The vegetation type detection model is represented as follows: Where N represents the total number of samples, This represents the probability that the current input sample belongs to the j-th category. This represents the true label of the current input sample belonging to the j-th vegetation type, where j={1,2,3}.

4. The vegetation type identification method based on the fusion of machine vision technology and deep learning according to claim 3, characterized in that, The activation function is the softmax activation function.

5. An electronic device, characterized in that, The device includes a processor and a memory, the memory being used to store one or more programs, the one or more programs including step instructions for a vegetation type identification method based on the fusion of machine vision technology and deep learning according to any one of claims 1-4, and the processor being used to execute the one or more programs.

6. A storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, can implement the steps of the vegetation type identification method based on the fusion of machine vision technology and deep learning according to any one of claims 1-4.

Images