Intelligent identification method, device and equipment for forest and fruit crops and storage medium

Abstract

The invention discloses an intelligent identification method, device and equipment for forest and fruit crops and a storage medium, and relates to the technical field of agricultural remote sensing information, and the method comprises the following steps: making land parcel information for a high-spatial-resolution image, and obtaining a plurality of corresponding land parcel labels; mapping the plurality of land parcel labels to corresponding multispectral images, and making land parcel crop type information for the mapped multispectral images to obtain a plurality of corresponding crop labels; training the BsiNet network through the plot data set to obtain a segmentation network; training the Hybrid SN network through the crop data set to obtain an identification network; and inputting a to-be-identified high spatial resolution image of the forest and fruit crop research area to the segmentation network to obtain corresponding land parcel information, and inputting the mapped to-be-identified multispectral image to the identification network to obtain corresponding land parcel crop type information. According to the method, crop identification is carried out by taking the land parcels as identification units, different features do not appear in the same area, and no classification noise exists.

Description

Technical Field

[0001] The present invention relates to the field of agricultural remote sensing information technology, and particularly to a method, device, equipment and storage medium for intelligent identification of forest and fruit crops. Background Art

[0002] Forest and fruit are important economic crops across the country. Precise crop mapping at a large scale can provide new data support for macro planting regulation. However, at present, in some areas, the semanticity of the intelligent remote sensing classification results of the spatial distribution of large-scale forest and fruit plantings is weak, the accuracy is not high, and the operation efficiency is low. Therefore, improving the accuracy and semanticity of intelligent forest and fruit classification, increasing the operation efficiency, and providing high-precision forest and fruit crop type information for smart agriculture and forest and fruit crop management are urgent problems to be solved in the current main battlefield of the forest and fruit economy.

[0003] Existing methods use artificial intelligence technology with pixels as the recognition unit to achieve the recognition of large-scale forest and fruit crops. This method directly performs intelligent recognition of forest and fruit crops through a deep network. This type of method does not require prior feature extraction, and the feature extraction and recognition processes are all completed by the deep network, with a high degree of intelligence.

[0004] In actual planting scenarios, the same type of forest and fruit crop is usually concentrated in a planting area for planting. However, the characteristics of different trees of the same type of forest and fruit in the same planting area on remote sensing images are not completely the same. When using artificial intelligence technology with pixels as the recognition unit to identify this area, different characteristics will appear in the same area, resulting in the problem of classification noise in the recognition results of this area, which is not conducive to the intelligent management of forest and fruit crops. Summary of the Invention

[0005] Based on the above-mentioned defects existing in the prior art, the present invention provides a method, device, equipment and storage medium for intelligent identification of forest and fruit crops, which solves the problem that different characteristics will appear in the same area when using the existing artificial intelligence technology with pixels as the recognition unit, resulting in the problem of classification noise in the recognition results of this area.

[0006] The present invention adopts the following technical solutions:

[0007] In the first aspect, the present invention provides a method for intelligent identification of forest and fruit crops, including the following steps:

[0008] Collect high-spatial-resolution images and multi-spectral images of the same period in the forest and fruit research area; the high-spatial-resolution images are images with a spatial resolution less than 5 meters;

[0009] Create plot information for the high-spatial-resolution images to obtain corresponding multiple plot labels; map the multiple plot labels to the corresponding multi-spectral images, and create plot crop type information for the mapped multi-spectral images to obtain corresponding multiple crop labels;

[0010] Taking the high-spatial-resolution image as the input and the corresponding multiple plot labels as the output, training the BsiNet network to obtain a segmentation network; taking the mapped multi-spectral image as the input and the corresponding multiple crop labels as the output, training the HybridSN network to obtain a recognition network;

[0011] Input the high-spatial-resolution image to be recognized in the fruit and forest crop research area into the segmentation network to obtain the corresponding plot information, map the plot information to the multi-spectral image to be recognized in the same period, and input the mapped multi-spectral image to be recognized into the recognition network to obtain the corresponding plot crop type information.

[0012] Preferably, training the BsiNet pre-trained network to obtain a segmentation network includes the following steps:

[0013] Obtain the BsiNet pre-trained network, and the model weight parameters in the BsiNet pre-trained network are the model weight parameters obtained by training the BsiNet network on the existing plot dataset;

[0014] Taking the high-spatial-resolution image as the input and the corresponding multiple plot labels as the output, training the BsiNet pre-trained network to obtain a segmentation network; wherein, during the training process, the weight parameters of the first two layers of the encoder of the BsiNet pre-trained network are frozen.

[0015] Preferably, training the BsiNet network to obtain a segmentation network includes the following steps:

[0016] Obtain the historical plot dataset, and pre-train the BsiNet network through the historical plot dataset to obtain the BsiNet pre-trained network;

[0017] Taking the high-spatial-resolution image as the input and the corresponding multiple plot labels as the output, training the BsiNet pre-trained network to obtain a segmentation network; wherein, during the training process, the weight parameters of the first two layers of the encoder of the BsiNet pre-trained network are frozen.

[0018] Preferably, use the SAM image large model to generate plot information for the high-spatial-resolution image.

[0019] Preferably, the multi-spectral image is an image with 4 or more spectral bands.

[0020] In a second aspect, the present invention provides an intelligent recognition device for fruit and forest crops, including:

[0021] The acquisition module is used to acquire high spatial resolution images of the fruit and forest research area and multi-spectral images of the same period; the high spatial resolution images are images with a spatial resolution of less than 5 meters;

[0022] A production module is used to produce plot information for high spatial resolution images to obtain multiple corresponding plot labels; map multiple plot labels to corresponding multispectral images, produce plot crop type information for the mapped multispectral images, and obtain multiple corresponding crop labels;

[0023] The training module is used to train the BsiNet network with high spatial resolution images as input and corresponding multiple plot labels as output to obtain a segmentation network; and to train the HybridSN network with mapped multispectral images as input and corresponding multiple crop labels as output to obtain a recognition network;

[0024] The recognition module is used to input the high spatial resolution image to be identified in the forestry and fruit crop research area into the segmentation network to obtain the corresponding plot information, map the plot information to the multispectral image to be identified in the same period, and input the mapped multispectral image to be identified into the recognition network to obtain the corresponding plot crop type information.

[0025] In a third aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned intelligent identification method for fruit crops when executing the program.

[0026] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned method for intelligent identification of fruit crops is implemented.

[0027] Compared with the prior art, at least one of the above technical solutions adopted by the present invention can achieve the following beneficial effects:

[0028] The present invention first collects high spatial resolution images of the forest and fruit research area and multispectral images of the same period, generates plot information for the high spatial resolution images, and obtains corresponding multiple plot labels. The BsiNet network is trained by the high spatial resolution images and the corresponding plot labels to obtain a segmentation network. Therefore, the present invention uses plots as identification units for crop identification, so that the identification results of the crop types of the same plot are the same, and different features will not appear in the same area. Further, multiple plot labels are mapped to corresponding multispectral images, and plot crop type information is generated for the mapped multispectral images to obtain corresponding multiple crop labels. The present invention uses multispectral images with rich spectral information to achieve high-precision crop identification. Specifically, the HybridSN network is trained by outputting the mapped multispectral images and multiple crop labels to obtain a recognition network. The present invention uses a network that organically integrates two different functions of plot segmentation and plot type identification, which can realize both intelligent segmentation of cultivated land plots and intelligent identification of plot crop types. Different features will not appear in the same area, and there is no classification noise in the identification results. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0030] Figure 1 A flow chart of a method for intelligent identification of fruit crops of the present invention;

[0031] Figure 2 The spatial distribution diagram of different areas of the present invention when the BsiNet land network is not used but only the HybridSN network is used;

[0032] in, Figure 2 (a): Spatial distribution map of plums in Barin Town, Jiashi County, Xinjiang. Figure 2 (b): Spatial distribution map of plums in Yingmaili Township, Jiashi County, Xinjiang;

[0033] Figure 3 The remote sensing thematic map of the spatial distribution of Ximei land in Baren Town, Jiashi County, Xinjiang, according to the present invention;

[0034] in, Figure 3 (a): Remote sensing thematic map of the spatial distribution of plums in Barin Town, Jiashi County, Xinjiang. Figure 3 (b): Remote sensing thematic map of the spatial distribution of plums in Yingmaili Township, Kashgar County, Xinjiang. DETAILED DESCRIPTION

[0035] Next, in combination with the accompanying drawings in the embodiments of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0036] I. Explanation and description of the embodiments. This part is an explanatory embodiment that expands and explains the technical solutions of the claims in order to enable those skilled in the art to fully understand how the present invention is specifically implemented.

[0037] To solve the technical problems existing in the prior art, referring to Figure 1 , the present invention provides an intelligent identification method for forest and fruit crops, including the following steps:

[0038] S1: Collect high-spatial-resolution images of the forest and fruit research area and multi-spectral images of the same period.

[0039] Collect multiple high-spatial-resolution images (such as GF-2 images, GF-1, GF-7, etc., high-resolution images better than 5 meters) of the research area at a given phenological period, as well as multi-spectral satellite images of the same period (such as Sentinel-2 images). Perform radiometric correction, orthorectification, and panchromatic and multi-spectral image mapping on the high-spatial-resolution images. Generally, images with a spatial resolution better than 5 meters are called high-spatial-resolution images. Generally, images with 4 or more spectral bands are called multi-spectral images.

[0040] Among them, high-spatial-resolution images have the advantage of plot segmentation. However, due to insufficient spectral information, it is impossible to accurately identify the crop categories in the plots. Therefore, it is necessary to use multi-spectral images with rich spectral information to achieve high-precision crop identification.

[0041] S2: Create plot information for the high-spatial-resolution images to obtain corresponding multiple plot labels; map the multiple plot labels to the corresponding multi-spectral images, and create plot crop type information for the mapped multi-spectral images to obtain corresponding multiple crop labels.

[0042] A method for assisting manual farmland plot annotation using the image large model SAM (Segmenting Anything Model) network. First, use the SAM pre-trained model to segment high-spatial-resolution images. Then, combined with the field crop type survey data, manually further modify the results for sample enhancement to enhance the generalization ability of the model. Finally, a plot dataset for training the segmentation network for plot segmentation is constructed. The plot dataset includes high-spatial-resolution images and corresponding multiple plot labels. The segmentation of plots mainly depends on the spatial texture information of the images. Therefore, only high-spatial-resolution images are used.

[0043] At the same time, map the multiple plot labels in the plot dataset to the corresponding multi-spectral images, make plot crop labels for the mapped multi-spectral images, and obtain a crop dataset for training the recognition network for crop recognition. The crop dataset includes the mapped multi-spectral images and corresponding multiple crop labels.

[0044] The recognition of crops mainly depends on the spectral information of the images. Therefore, multi-spectral images need to be used.

[0045] S3: Train the BsiNet network through the plot dataset to obtain a segmentation network.

[0046] Using high-spatial-resolution images such as GF2 as the main data source, and using the plot dataset constructed in the early stage, retrain the BsiNet pre-trained network using the idea of transfer learning to obtain a trained BsiNet network model, and finally perform intelligent segmentation of cultivated land plots. The specific method is as follows:

[0047] 1) Use the model weight parameters obtained by training the BsiNet network on other large GF-2 plot datasets as the pre-trained model.

[0048] 2) During the training of the plot segmentation task of the present invention, freeze the weight parameters of the first two layers of the encoder in the BsiNet pre-trained weights, and train other layers of the model on this basis to accelerate the convergence process of the network and improve performance. Use high-spatial-resolution images as the input and corresponding multiple plot labels as the output to train the BsiNet pre-trained network.

[0049] 3) In terms of the loss function, the negative log-likelihood loss function is used for the farmland plot prediction and boundary prediction tasks, and the mean square error loss function is used for the boundary distance prediction task.

[0050] S4: Train the HybridSN network through the crop dataset to obtain a recognition network.

[0051] With the mapped multispectral image as input and the corresponding multiple crop labels as output, the HybridSN network is trained to obtain the recognition network.

[0052] S5: Input the high spatial resolution image to be identified in the fruit crop study area into the segmentation network to obtain the corresponding plot information, map the plot information to the multispectral image to be identified in the same period, input the mapped multispectral image to be identified into the recognition network to obtain the corresponding plot crop type information.

[0053] 2. Evidence of the effects of the embodiments The embodiments of the present invention have achieved some positive effects during the development or use process, and indeed have great advantages over the prior art, which are described below in conjunction with the data and charts of the test process.

[0054] In order to verify the feasibility of the method of the present invention, the western plum orchard in Baren Town, Kashgar County, Xinjiang, was taken as an example, and the high spatial resolution images such as GF2 were used as the data source, and the BsiNet pre-trained network was used to segment the cultivated land. The farmland plot dataset was obtained, the spatial positions of the high-resolution image and the multispectral image were unified (mapped), and the registered multispectral image was used as the data source. The prepared western plum plot dataset was used, and the western plum plot was identified using the HybridSN network to obtain the western plum orchard planting plot dataset.

[0055] The method proposed in the present invention is applied to Baren Town, Kashgar County, Xinjiang, and other places, and verified with field survey data. The two indicators mIoU (Mean Intersection over Union) and F1-Score are used to evaluate the accuracy of plot extraction. Among them, mIoU is a commonly used semantic segmentation indicator, which is used to measure the classification accuracy of the segmentation network for each category. mIoU takes into account the degree of overlap between the predicted results and the true labels of each category, and then takes the average of all categories. The closer this value is to 1, the better the classification effect of the segmentation network for each category. The F1-Score indicator combines the information of accuracy and recall rate, and evaluates the comprehensive performance of the network by harmonizing the mean. Its value range is also between 0 and 1. The closer the value is to 1, the better the balance between the positive and negative examples of the segmentation network, and the better the network prediction accuracy. The accuracy evaluation of the plot data extraction results in Baren Town, Kashgar County, Xinjiang, carried out using the above indicators is shown in Table 1.

[0056] Table 1 Evaluation of data segmentation accuracy of plots in Barin Town and Yingmaili Township, Kashgar County, Xinjiang

[0057]

[0058] It can be seen from Table 1 that the mIoU value for Barin Town is 0.84, and the F1-score value is 0.86; the mIoU value for Yingmaili Township is 0.89, and the F1-score value is 0.94, which indicates that the segmentation network of the present invention has a high accuracy in segmenting farmland plots and a good accuracy in extracting cultivated land plots.

[0059] The performance evaluation indexes of the recognition result of the western plum plot of the present invention are the overall accuracy (OA), average accuracy (AA) and Kappa coefficient. The recognition accuracy evaluation of the western plum plot in Baren Town, Jiashi County, Xinjiang, carried out using the above indicators is shown in Table 2.

[0060] Table 2 Evaluation of recognition accuracy of Ximei plots in Baren Town and Yingmaili Township, Kashgar County, Xinjiang

[0061]

[0062] It can be seen from Table 2 that the overall accuracy (Overall Accuracy) of the present invention for identifying the plum plots in Barin Town, Kashgar County, Xinjiang is 99.27%, the average accuracy is 97.47%, and the Kappa coefficient is 96.29%. The overall accuracy (Overall Accuracy) of the present invention for identifying the plum plots in Yingmaili Township is 97.98%, the average accuracy is 97.52%, and the Kappa coefficient is 95.45%, which verifies that the present invention has high accuracy in identifying the plum plots in the region.

[0063] In order to further evaluate the performance of the recognition network, the present invention compares the classification results of the recognition network with the official statistics of prune planting data. As shown in Table 3, through this comparison, the consistency of the classification results of the recognition network and the actual data can be evaluated, thereby verifying its accuracy and reliability in practical applications. As an external standard, the official data provides an objective reference benchmark, which helps to verify the practical value and actual effect of the recognition network.

[0064] Table 3 Comparison of the number of acres of prunes identified and official statistics

[0065]

[0066] Taking the intelligent recognition of plum fruit crops in Baren Town, Jiashi County, Xinjiang as an example, the recognition results of plum crops using the HybridSN model alone are as follows: Figure 2 (a) and Figure 2 As shown in (b). Figure 2 (a) and Figure 2As can be seen from (b) of [reference], the results extracted by using the HybridSN model alone show pixel-level discreteness, making it difficult to understand the crop distribution of plots as a whole. Moreover, due to the resolution limitation of Sentinel-2 data, the results also show obvious jaggedness, which is not conducive to the information discrimination and secondary utilization of subsequent data results.

[0067] The space distribution map of plum crops produced by the present invention is as shown in Figure 3 (a) of [reference] and Figure 3 (b) of [reference]. As can be seen from Figure 3 (a) of [reference] and Figure 3 (b) of [reference], the method for identifying and extracting plum crops that integrates the BsiNet and HybridSN models identifies plum crops in units of plots. The results are plot-based and can fully reflect the crop information within the plots. It should be noted that the extraction results are not limited by the spatial resolution of multi-spectral and hyper-spectral images, better meeting the requirements for crop information extraction. By identifying crops in units of plots, the spatial distribution and correlation of crops within the plots can be captured more accurately, and then more continuous and accurate crop extraction results can be obtained. Finally, there is no classification noise in the output planting space distribution map of this cultivated land plot.

[0068] Through experimental verification, the use of the present invention is consistent with the results of on-site investigation and manual visual interpretation for the spatial distribution of plum crop plots. In Baren Town, the upper part of the plum planting area is mainly distributed in its eastern part, and the planting area is relatively concentrated, occupying a considerable proportion in the entire cultivated land area.

[0069] The present invention is extended to the tasks of segmenting and identifying plots of forest and fruit crops such as plums in other regions such as Yingmaili Township, Jiashi County, Xinjiang, verifying the applicability and accuracy of the present invention, and having practical application and promotion value.

[0070] The data sources (materials) used in the present invention are from domestic optical satellite images with a resolution of one meter or free multi-spectral images with medium and high spatial resolutions. The data acquisition cost is low, and the advantages of the two-source images in spatial resolution and spectral resolution are complementary.

[0071] The present invention organically integrates models with two different functions of plot segmentation and plot type identification, can not only realize the intelligent segmentation of cultivated land plots) and the intelligent identification of plot crop types, and can output the planting space distribution map of this cultivated land plot.

[0072] The present invention can be applied to the intelligent identification task of forest and fruit plots on a large scale (such as the scale of Jiashi County, Xinjiang with an area of 6715.4 square kilometers).

[0073] The present invention can produce vector and raster remote sensing thematic maps of cultivated land plots, and can also identify the crop types of the plots, and the correct recognition rate of the plots is above 90%.

[0074] The present invention is not only applicable to the recognition of the planting types of single-crop plots, but also can be directly applied to the classification and recognition of the planting types of multi-crop plots.

[0075] Based on the same concept, the present invention also provides an intelligent recognition device for forest and fruit crops, including a collection module, a production module, a training module and an identification module.

[0076] The collection module is used to collect high-spatial-resolution images of the forest and fruit research area and multi-spectral images of the same period; the high-spatial-resolution images are images with a spatial resolution less than 5 meters.

[0077] The production module is used to produce plot information from the high-spatial-resolution images to obtain corresponding multiple plot labels; map the multiple plot labels to the corresponding multi-spectral images, and produce plot crop type information from the mapped multi-spectral images to obtain corresponding multiple crop labels.

[0078] The training module is used to train the BsiNet network with the high-spatial-resolution images as the input and the corresponding multiple plot labels as the output to obtain a segmentation network; train the HybridSN network with the mapped multi-spectral images as the input and the corresponding multiple crop labels as the output to obtain an identification network.

[0079] The identification module is used to input the high-spatial-resolution image to be identified in the forest and fruit crop research area into the segmentation network to obtain corresponding plot information, map the plot information to the multi-spectral image to be identified in the same period, and input the mapped multi-spectral image to be identified into the identification network to obtain corresponding plot crop type information.

[0080] The present invention also provides a computer device, including a memory, a processor and a computer program stored on the memory and executable on the processor. When the processor executes the program, the above-mentioned intelligent recognition method for forest and fruit crops is realized.

[0081] The present invention also provides a computer-readable storage medium, and the storage medium stores a computer program. When the computer program is executed by the processor, the above-mentioned intelligent recognition method for forest and fruit crops is realized.

[0082] Although the preferred embodiments of the present invention have been described, those skilled in the art can make additional changes and modifications to these embodiments once they know the basic creative concept. Therefore, the appended claims are intended to be construed as including the preferred embodiments and all changes and modifications falling within the scope of the present invention.

[0083] Obviously, those skilled in the art can make various modifications and variations to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalent technologies, the present invention also intends to include these modifications and variations.

Claims

1. An intelligent recognition method for forest and fruit crops, characterized in that, Including the following steps: Collect high-spatial-resolution images of the forest fruit research area and multi-spectral images of the same period; the high-spatial-resolution images are images with a spatial resolution less than 5 meters; Make plot information for the high-spatial-resolution images to obtain corresponding multiple plot labels; map the multiple plot labels to the corresponding multi-spectral images, and make plot crop type information for the mapped multi-spectral images to obtain corresponding multiple crop labels; Use the high-spatial-resolution images as input and the corresponding multiple plot labels as output to train the BsiNet network to obtain a segmentation network; Use the mapped multi-spectral images as input and the corresponding multiple crop labels as output to train the HybridSN network to obtain an identification network; Input the high-spatial-resolution images to be identified in the forest fruit crop research area into the segmentation network to obtain corresponding plot information, map the plot information to the multi-spectral images to be identified in the same period, and input the mapped multi-spectral images to be identified into the identification network to obtain corresponding plot crop type information.

2. The intelligent recognition method for forest and fruit crops according to claim 1, wherein The training of the BsiNet network to obtain a segmentation network includes the following steps: Obtain a historical plot dataset and pre-train the BsiNet network through the historical plot dataset to obtain a BsiNet pre-trained network; Use the high-spatial-resolution images as input and the corresponding multiple plot labels as output to train the BsiNet pre-trained network to obtain a segmentation network; among them, during the training process, freeze the weight parameters of the first two layers of the encoder of the BsiNet pre-trained network.

3. The intelligent recognition method for forest fruits crops according to claim 1, wherein, Use the SAM image large model to make plot information for the high-spatial-resolution images.

4. The intelligent recognition method for forest and fruit crops according to claim 1, characterized in that, The multi-spectral images are images with 4 or more spectral bands.

5. An intelligent recognition device for forest and fruit crops, characterized in that, Including: A collection module for collecting high-spatial-resolution images of the forest fruit research area and multi-spectral images of the same period; the high-spatial-resolution images are images with a spatial resolution less than 5 meters; A production module for making plot information for the high-spatial-resolution images to obtain corresponding multiple plot labels; mapping the multiple plot labels to the corresponding multi-spectral images, and making plot crop type information for the mapped multi-spectral images to obtain corresponding multiple crop labels; A training module for using the high-spatial-resolution images as input and the corresponding multiple plot labels as output to train the BsiNet network to obtain a segmentation network; Use the mapped multi-spectral images as input and the corresponding multiple crop labels as output to train the HybridSN network to obtain an identification network; An identification module for inputting the high-spatial-resolution images to be identified in the forest fruit crop research area into the segmentation network to obtain corresponding plot information, mapping the plot information to the multi-spectral images to be identified in the same period, and inputting the mapped multi-spectral images to be identified into the identification network to obtain corresponding plot crop type information.

6. A computer device, characterized in that, Including a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the program, it implements the intelligent identification method for forest fruit crops described in any one of claims 1-4 above.

7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and when the computer program is executed by a processor, it implements the intelligent identification method for forest fruit crops described in any one of the above claims 1-4.

Images