Method for identifying planting distribution of crops based on deep learning and fusion positioning technology

Abstract

The invention provides a crop planting distribution identification method based on deep learning and a fusion positioning technology. The method comprises the steps of 1, obtaining a camera image, a three-dimensional point cloud and a geographic coordinate of a target area through an acquisition device; step 2, identifying crops in the camera image through a deep learning algorithm, identifying the boundary of each crop planting area at the same time, and outputting crop classification vector pattern spots; step 3, projecting radar point cloud data to a camera image by adopting a PCL, fusing the camera image and the three-dimensional point cloud to obtain an accurate coordinate of each pixel point in the camera image relative to the surveying and mapping point, and outputting a pixel coordinate matrix; and 4, calculating the area of each crop distribution region according to the coordinates of the pixel points on the boundary of each distribution region in the crop classification vector pattern spots, summarizing the same crops, and finally counting the planting area and the distribution condition of each crop.

Description

Technical Field

[0001] The present application relates to the field of agricultural informatization, and in particular to a method for identifying the planting distribution of crops based on deep learning and fusion positioning technology. Background Art

[0002] Currently, for the statistical calculation of the planting areas in each region of crop distribution, mainly various planters (including farmers and large-scale planters) report data, and then it is subject to multi-level review by villages, townships, and counties. The basis for review mostly utilizes satellite remote sensing technology to process, train, model, and identify multi-spectral or hyperspectral remote sensing image data, and finally calculates the data results.

[0003] However, affected by the number of satellites and their operating cycles, there will be a lack of remote sensing data in some periods of the whole year. Moreover, weather is also an uncontrollable factor. In some areas with long-term rain or cloud cover, some parts of the remote sensing area will be covered by clouds.

[0004] In addition, the higher the precision of remote sensing data, the more expensive its purchase cost. For all these reasons, satellite remote sensing technology has some serious problems that urgently need to be solved in crop identification.

[0005] Of course, using drones for low-altitude remote sensing operations to assist in solving the above problems of satellite remote sensing is indeed a good solution. However, the disadvantages of low-altitude remote sensing operations are also very obvious, that is, the operations are also affected by weather and personnel, and the input cost is relatively high, and the acquisition and processing efficiency is relatively low. Summary of the Invention

[0006] In order to solve at least one of the technical problems in the background art, the embodiments of the present application propose a method for identifying the planting distribution of crops based on deep learning and fusion positioning technology, which is based on satellite positioning, radar scanning, camera photography, and artificial intelligence technology. Through the high-resolution images obtained by camera photography and combined with relevant deep learning models, the rapid identification of various crops in the photos can be achieved. At the same time, the coordinates of the current shooting point are obtained through the satellite, and the laser point cloud data obtained by the radar device are fused to quickly calculate the accurate coordinates of each pixel point in the image. Finally, the distribution areas of various crops in the photo can be statistically calculated. This method can greatly reduce the cost of collecting data and significantly improve the intelligent level of agricultural production verification.

[0007] The above method for identifying the planting distribution of crops based on deep learning and fusion positioning technology includes the following steps:

[0008] Step 1: Obtain the camera image, three-dimensional point cloud, and geographical coordinates of the target area through the acquisition device;

[0009] Step 2: Identify the crops in the camera images through a deep learning algorithm, and at the same time identify the boundaries of the planting areas of each crop, and output a classified vector map of the crops;

[0010] Step 3: Use PCL to project the lidar point cloud data onto the camera image, fuse the camera image with the 3D point cloud, obtain the accurate coordinates of each pixel point in the camera image relative to the survey point, and output a pixel coordinate matrix;

[0011] Step 4: Calculate the area of each crop distribution area based on the coordinates of the pixel points on the boundaries of the distribution areas in the classified vector map of the crops, summarize the same crops, and finally count the planting areas and distribution conditions of each crop.

[0012] In one embodiment of the present application, for the method for identifying the planting distribution of crops based on deep learning and fusion positioning technology, Step 2 includes the following:

[0013] Step 2.1: Select a binary-based classifier, regard each label as an independent binary classification problem, and achieve pixel-level multi-label classification through end-to-end training;

[0014] Step 2.2: According to the phenological map, collect field photos of each region, each season, and each weather condition. For the collected image samples, perform image enhancement processing, as well as image cropping and mosaicking;

[0015] Step 2.3: Label the samples. After labeling the images, the distribution areas of each crop and the negative list can be seen. The negative list includes wasteland, pond, greenhouse, orchard, vegetable plot, and grassland;

[0016] Step 2.4: The model configuration uses SegNeXt based on the convolutional attention architecture scheme, as well as the multi-scale convolutional attention module MSCAN_L;

[0017] Step 2.5: Model construction and prediction;

[0018] Step 2.6: The metrics used for model evaluation and hyperparameter tuning include mean average precision, precision, and recall. Repeatedly adjust the learning rate and batch size hyperparameters until the target training effect is achieved;

[0019] Step 2.7: When the model prediction accuracy reaches the target training effect, the shp vector file output by the model is the final result, that is, the classified vector map of the crop distribution, and the classified vector map of the crop distribution indicates the distribution areas of each crop.

[0020] In one embodiment of the present application, for the method for identifying the planting distribution of crops based on deep learning and fusion positioning technology, the image enhancement processing in Step 2.2 includes the following:

[0021] For those with overly concentrated and small-contrast random distributions of pixel brightness, adjust them to a normal distribution; classify the gray values of gray-scale image pixels and assign different colors; stretch the gray-scale range of the image to the saturated state in the range of 0 to 255, and then swap the positive and negative images; perform noise removal, image smoothing, and sharpening operations on the pixels in each region using a 5x5 filter.

[0022] In one embodiment of the present application, the method for identifying the planting distribution of crops based on deep learning and fusion positioning technology, the cropping and mosaicking of the images in step 2.2 include the following:

[0023] Image fusion; retain the regions of interest such as crop planting areas and remove excessive sky and ground backgrounds; crop according to the longitude and latitude ranges of the geographical coordinates; use the deep learning imagededup library to remove duplicates from multiple images taken around the target point and then stitch them into a whole to describe the planting situation in the current region.

[0024] In one embodiment of the present application, the method for identifying the planting distribution of crops based on deep learning and fusion positioning technology, step 3 includes the following:

[0025] Step 3.1: Use PCL to project the radar point cloud data onto the camera image. The PCL is a function library for processing three-dimensional point cloud data, including point cloud filtering, registration, segmentation, and feature extraction;

[0026] Step 3.2: Perform georegistration and projection correction on the image and point cloud data to ensure the accuracy and consistency of the data, and process the conversion between the radar coordinate system and the camera coordinate system. To convert the radar coordinate system to the camera coordinate system, it is necessary to rotate counterclockwise 90 degrees around the x-axis, then rotate clockwise 90 degrees around the y-axis, and rotate 0 degrees around the z-axis. Its rotation matrix is:

[0027]

[0028] Step 3.3: Implement the above conversion from the radar coordinate system to the camera coordinate system using C++ code, and finally project all the radar point cloud data onto the camera image. The algorithm is as follows:

[0029] Define a structure that contains the topics of the camera and lidar, the camera internal parameter matrix camIn, and the RT matrix;

[0030] Initialize the structure, read the topic names of the camera and lidar, the camera internal parameter matrix, and the RT matrix and other parameters from the configuration file, and store them in the corresponding variables of the structure iPara;

[0031] Fuse the camera image and radar point cloud data. First, create two message filters to subscribe to the camera image and lidar point cloud data respectively. Then, use the ApproximateTime strategy for synchronization. Next, create an image transport object and publish the processed image data. Finally, enter the message loop to project the received point cloud data onto the image for fusion;

[0032] Step 3.4: Convert the image data in the message to the Mat format of OpenCV and convert the lidar point cloud data to pcl::PointCloud <pcl::pointxyzi>The format is stored in the cloud, and then the original image data is copied into visImg. The lidar point cloud data is traversed, and the following processing is performed on each point:

[0033] If the x - coordinate of the point is greater than maxX or less than 0, the absolute value of the y - coordinate is greater than maxY, or the z - coordinate is less than minZ, then skip this point;

[0034] Store the x, y, z coordinates of the point and 1 into X;

[0035] Calculate the fused point coordinates Y = camIn * RT * X;

[0036] Map the Y - coordinates to the image coordinate system, calculate the color according to the x - value of the point, and finally draw a circle on the overlay image;

[0037] After processing all the point cloud data, set the image encoding format to bgr8. Finally, publish the processed image data through image_publisher;

[0038] Step 3.5: Each point data in the obtained three - dimensional point cloud data structure pcl::PointXYZI corresponds to the three - dimensional coordinates of a pixel point in the image, that is, the pixel coordinate matrix. Traverse the point cloud data and save the data to a csv file to provide a basis for the next step of area calculation.

[0039] In one embodiment of the present application, for the method of identifying the planting distribution of crops based on deep learning and fusion positioning technology, the step 4 includes the following:

[0040] Step 4.1: Read the crop distribution vector map patch file to obtain the planting areas of various crops; Use the geopandas library to read the previously stored shp file, extract the coordinate points of the outer contour of each polygon, and calculate its serial number from the inflection point coordinates of the polygon;

[0041] Step 4.2: Read the pixel coordinate matrix file to obtain the physical coordinates corresponding to the inflection points of each polygon in the vector map patch;

[0042] Step 4.3: Calculate the area through the coordinates of each point of the polygon, summarize the polygon areas according to the crop planting types, and then the categories and planting areas of the crops around the measurement points can be calculated.

[0043] In one embodiment of the present application, for the method of identifying the planting distribution of crops based on deep learning and fusion positioning technology, the acquisition device in the step 1 includes a satellite positioning device, a radar, and a camera.

[0044] Advantages of the present application:

[0045] 1. Supplementary interpretation technology of auxiliary satellite remote sensing for full-coverage, unobstructed, and dead-angle-free: Solve the problem that due to the influence of weather or operation cycle on satellites, remote sensing images are missing in a certain administrative region, or the coverage is incomplete, or some areas are covered by clouds. At this time, using ground equipment for data collection can be said to be an almost perfect solution.

[0046] 2. Small-scale image acquisition can completely replace low-altitude remote sensing operations of drones. For example, when conducting crop verification in a certain village or investigating arable land abandonment in several pilot townships, data analysis and statistics can be carried out by uploading photos through ground equipment in the target area. At the same time, the photos can also be used as evidence for on-site comparison.

[0047] 3. Adding an edge computing module to the ground equipment can quickly obtain the planting situation of crops in remote mountainous areas and complex environment areas without the need for a network, truly achieving rapid calculation, knowing at a glance with one shot, and knowing immediately with each shot. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0049] Figure 1 It is a schematic diagram of the overall technical process of the method for identifying the planting distribution of crops based on deep learning and fusion positioning technology provided by the present application;

[0050] Figure 2 It is a camera image of the method for identifying the planting distribution of crops based on deep learning and fusion positioning technology provided by the present application;

[0051] Figure 3 It is a vector file output after being processed by the deep learning algorithm of the method for identifying the planting distribution of crops based on deep learning and fusion positioning technology provided by the present application;

[0052] Figure 4 It is a schematic diagram of the radar coordinate system;

[0053] Figure 5 It is a schematic diagram of the camera coordinate system;

[0054] Figure 6 It is an image after fusing the camera image and the laser point cloud data of the method for identifying the planting distribution of crops based on deep learning and fusion positioning technology provided by the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0055] Currently, the statistical work of the planting areas in each region of crop distribution mainly relies on data reports from various planters (including farmers and large-scale planters), which are then subject to multi-level reviews at the village, township, and county levels. The basis for the review mostly utilizes satellite remote sensing technology to process, train, model, and identify multi-spectral or hyperspectral remote sensing image data, and finally calculates the data results. However, affected by the number of satellites and their operating cycles, there will be periods of missing remote sensing data throughout the year. Weather is also an uncontrollable factor. In some areas with long-term rain or cloud cover, some parts of the remote sensing area will be covered by clouds. Additionally, the higher the precision of the remote sensing data, the more expensive its purchase cost. For all these reasons, satellite remote sensing technology has some serious problems that urgently need to be solved in crop identification. Of course, using drones for low-altitude remote sensing operations to assist in solving the above problems of satellite remote sensing is indeed a good solution. However, the disadvantages of low-altitude remote sensing operations are also very obvious, that is, the operations are also affected by weather and personnel, and the input cost is relatively high, and the collection and processing efficiency is relatively low.

[0056] With the development of high-precision Beidou satellite positioning, lidar target measurement, and artificial intelligence technology, additional technical means are provided for the collection, identification, and analysis of crops. The requirements for collecting data can be further reduced. The low-altitude operation can be changed to ground operation, the spectral camera can be changed to a combination of an ordinary camera and lidar, and professional personnel can be changed to ordinary personnel. Then, data can be collected anytime and anywhere, and the operation cost of collection can be reduced to an extremely low level; the collection efficiency also increases geometrically, and the collected data can be immediately uploaded to the server or edge for processing, and then the crop types in various planting areas in the photo can be quickly identified.

[0057] In summary, combining satellite positioning technology, radar, and artificial intelligence technology can replace low-altitude remote sensing operations and assist satellite remote sensing interpretation, providing a powerful means for the statistics of large-scale crop distribution.

[0058] I. The technical problems to be solved in the embodiments of the present application are as follows:

[0059] 1. Classify and identify crops in ordinary photos:

[0060] Diversity problem: The photo may contain multiple crop and other land cover types, and the model needs to have strong generalization ability to accurately classify.

[0061] Data quality problem: Factors such as the shooting point, angle, height, and focal length of the photo will affect the quality of the photo, which may lead to problems such as incomplete annotation and unbalanced samples, and a large amount of data needs to be cleaned and processed.

[0062] Annotation accuracy problem: The annotation must be accurate and error-free, and a sufficient number of samples are required for training to improve the recognition accuracy of the model.

[0063] Type confusion problem: Features with the same color but different types need to be processed by comparing the local and the whole to avoid recognition errors such as inaccurate types and fragmented regions.

[0064] 2. Fusion of photos and point clouds:

[0065] Data fusion algorithm: An efficient and accurate data fusion algorithm needs to be developed to effectively fuse information from different data sources and ensure the accuracy of the final fusion result.

[0066] Data registration: Achieve accurate data registration to align the photos and the lidar point cloud in space to ensure their consistency in the same coordinate system.

[0067] Perspective correction: Perform perspective correction on the photos to convert them into orthophotos for matching and fusing with the lidar point cloud.

[0068] Noise processing: Process the noise in the lidar point cloud data, remove interference factors, and improve the data quality and stability.

[0069] Point cloud imaging: Achieve effective imaging of the lidar point cloud data, ensure the integrity and accuracy of the point cloud data, and avoid problems such as jitter during mapping.

[0070] Data fusion accuracy: The fusion result is required to reach the pixel level of accuracy to ensure the accuracy and integrity of the fusion of photo and lidar point cloud data.

[0071] II. Technical solutions adopted to solve the technical problems:

[0072] 1. Overall technical process:

[0073] The overall technical process is as Figure 1 shown. The ground acquisition equipment mainly includes satellite positioning, radar, and camera. When the equipment is started and the shooting is clicked, three types of data will be collected simultaneously.

[0074] 1) High-definition photos: Obtained through camera surveying. The shooting range, angle, and height are the same as those of the radar. The camera can provide a very high spatial resolution, which is especially suitable for capturing surface features with rich details. It can capture rich color and texture information, which is very useful for image recognition and classification.

[0075] 2) 3D point cloud: Obtained through lidar. The radar is not affected by weather and lighting conditions and can obtain data under various environmental conditions. The radar wave can penetrate tree canopies, vegetation, and other coverings to directly measure the surface information. The obtained 3D point cloud is a collection of the 3D spatial information of crops. Each point contains its position coordinates in the 3D space and possibly other attributes, such as color, reflectivity, etc.

[0076] 3) Satellite positioning: Through the GNSS (Global Navigation Satellite System) module, the precise coordinates of the current measurement point can be obtained in real time. This patent uses a BDS (Beidou Satellite Navigation) chip, which is a global satellite navigation system independently built and operated by our country and can provide all-weather, all-time, high-precision positioning, navigation, and timing services for global users.

[0077] After image recognition of high-definition photos, the planting distribution of crops in each area can be accurately predicted, and a classified vector map of crops can be output. After the high-definition photos are fused with the three-dimensional point cloud, the precise coordinates of each pixel point in the photo relative to the survey point can be calculated, and a pixel coordinate matrix can be output. According to the coordinates of the pixel points on the boundaries of each distribution area in the classified vector map of crops, the area of each distribution area can be calculated, and the same crops can be summarized to finally count the planting areas of various types of crops.

[0078] 2. Image recognition algorithm

[0079] The recognition algorithm mainly uses deep learning algorithms to identify as many crops as possible from the images taken by the camera, such as rice, lotus root, tobacco leaves, loofah, cotton, wheat, corn, vegetables, etc., and includes the boundaries of the planting areas of each crop. The relevant algorithms and code implementations are as follows:

[0080] 1) Model selection: A binary-based classifier can be selected, considering each label as an independent binary classification problem, such as Support Vector Machine (SVM), Logistic Regression, and Multi-Layer Perceptron, etc. Conditional Random Field or graph models can also be used to capture the joint probability distribution between labels. In this invention, a deep learning Convolutional Neural Networks (CNNs) model is selected, which can achieve pixel-level multi-label classification through end-to-end training.

[0081] 2) Data collection and cleaning: According to the phenological map, field photos of various regions, seasons, and weathers are collected. To balance the samples, the distribution of the collected data training samples is as follows in the table:

[0082]

[0083] The initial total sample size is:

[0084] L = 12×10×3×5×5×5 = 45000 (pieces)

[0085] 3) For the collected image samples, perform image enhancement processing:

[0086] Histogram transformation: For pixels with too concentrated random brightness distribution and low contrast, adjust them to normal distribution;

[0087] Density segmentation: the grayscale values of grayscale image elements are graded and assigned different colors;

[0088] Grayscale inversion: stretch the grayscale range of the image to the range of 0 to 255 to saturation, and then swap the positive and negative images;

[0089] Neighborhood enhancement: denoising, image smoothing, sharpening, etc. are performed on pixels in a certain area using a 5x5 filter;

[0090] Crop and mosaic an image:

[0091] Image Fusion:

[0092] Crop by ROI: that is, only keep the area of interest, such as the crop planting area, and remove the excessive sky and ground background;

[0093] Clipping by map: Clipping according to the latitude and longitude range of geographic coordinates;

[0094] Image stitching: With the target point as the center, multiple images taken around the target point are stitched together into a whole to describe the planting conditions in the current area after deduplication using the deep learning imagededup library.

[0095] 4) Sample labeling: For small sample sizes, you can use LabelImg or Labelme for local labeling software, and for tens of thousands of samples, you can use X-Anylabeling for automatic labeling. Visual tasks support image classification, target detection, instance segmentation, etc.; polygons, rectangles, rotation boxes, circles, lines, points, etc. can be labeled. After the images are labeled, you can see the distribution area of each crop and the negative list, where the negative list includes abandoned land, ponds, greenhouses, orchards, vegetable fields, grasslands, etc.; for the background such as the sky, houses, and villages, no labeling is required.

[0096] 5) Model configuration: In terms of model structure, SegNeXt based on the convolutional attention architecture and the multi-scale convolutional attention module MSCAN_L are used. Some parameters of its model configuration file segnext_msc an_l_crop.yaml are as follows:

[0097]

[0098]

[0099] 6) Model construction and prediction, the core code is as follows:

[0100]

[0101]

[0102]

[0103] 7) Model evaluation and hyperparameter tuning: Evaluate using metrics such as mAP (mean average precision), Precision, and Recall. Hyperparameters such as the learning rate and batch size need to be repeatedly adjusted to obtain the best training effect.

[0104] 8) Vector patches of crop distribution: When the model prediction accuracy reaches the optimal effect, the shp vector file output by the model is the final result, that is, the vector patches of crop distribution, as shown in Figure 2 and Figure 3 This vector file indicates the distribution areas of various crops and provides a basis for calculating the planting area later.

[0105] 3. Fusion algorithm

[0106] The fusion algorithm mainly uses PCL to project radar point cloud data onto the camera image. Here, PCL is the PointCloud Library, a function library for processing 3D point cloud data, including point cloud filtering, registration, segmentation, feature extraction, etc.

[0107] First, perform georegistration and projection correction on the image and point cloud data to ensure the accuracy and consistency of the data. Then, handle the conversion between the radar coordinate system and the camera coordinate system. Their coordinate systems are as shown in Figure 4 and Figure 5 where:

[0108] Radar coordinate system: x forward, y left, z up;

[0109] Camera coordinate system: z forward, x right, y down;

[0110] From Figure 4 and Figure 5 it can be seen that to convert the radar coordinate system to the camera coordinate system, it is necessary to rotate 90 degrees counterclockwise around the x-axis, then rotate 90 degrees clockwise around the y-axis, and rotate 0 degrees around the z-axis. Its rotation matrix is:

[0111]

[0112]

[0113] Implement the above conversion from the radar coordinate system to the camera coordinate system using C++ code. Finally, project all the radar point cloud data onto the camera image. The key algorithms and code implementation are as follows:

[0114] 1) Define a structure. It contains topics of the camera and lidar, the camera internal parameter matrix camIn, and the RT matrix.

[0115]

[0116] 2) Initialize the structure. Read the topic names of the camera and lidar, parameters such as the camera internal parameter matrix and the RT matrix from the configuration file, and store them in the corresponding variables of the structure iPara.

[0117] 3) Fuse the camera image and lidar point cloud data. First, create two message filters to subscribe to the camera image and lidar point cloud data respectively. Then use the ApproximateTime policy for synchronization. Then create an image transport object and publish the processed image data. Finally, enter the message loop and project the received point cloud data onto the image for fusion.

[0118] projection(){

[0119] message_filters::Subscriber<sensor_msgs::Image> imgSub(nh, iPara.camTopic, 30);

[0120] message_filters::Subscriber<sensor_msgs::PointCloud2> lidarSub(nh, iPara.lidarTopic, 10);

[0121] typedef message_filters::sync_policies::ApproximateTime<sensor_msgs::Image, sensor_msgs::PointCloud2> SyncPolicy;

[0122] message_filters::Synchronizer <syncpolicy>sync(SyncPolicy(5), imgSub, lidarSub);

[0123] sync.registerCallback(boost::bind(&projection::projection_callback, this, _1, _2));

[0124] image_transport::ImageTransport imageTranspors(nh);

[0125] image_publisher = imageTranspors.advertise(" / prj_img", 20);

[0126] }

[0127] 4) In the previous projection_callback function, the fusion process of lidar point cloud data and camera images was implemented, and the processed image data was published. Inside the function, first, the image data in the message was converted into the Mat format of OpenCV, and the lidar point cloud data was converted into pcl::PointCloud <pcl::pointxyzi>Format, stored in the cloud. Then copy the original image data to visImg, traverse the lidar point cloud data, and perform the following processing on each point:

[0128] - If the x - coordinate of the point is greater than maxX or less than 0, the absolute value of the y - coordinate is greater than maxY, or the z - coordinate is less than minZ, skip the point.

[0129] - Store the x, y, z coordinates of the point and 1 into X.

[0130] - Calculate the fused point coordinates Y = camIn * RT * X.

[0131] - Map the Y coordinates to the image coordinate system, calculate the color according to the x value of the point, and finally draw a circle on the overlay image.

[0132] After processing all the point cloud data, set the image encoding format to bgr8. Finally, publish the processed image data through the image_publisher, see Figure 6 。

[0133]

[0134]

[0135] 5) Pixel coordinate matrix: The three - dimensional point cloud data structure pcl::PointXYZI obtained in the previous step, where the data of each point corresponds to the three - dimensional coordinates of a pixel point in the image, that is, the pixel coordinate matrix. Traverse the point cloud data, first traverse the rows and then the columns, and save the data to a csv file for providing a basis for the next step of area calculation.

[0136]

[0137] 4. Crop distribution statistics and evaluation

[0138] 1) Read the crop distribution vector map patch file to obtain the planting areas of various crops. Use the geopandas library to read the previously stored shp file, extract the geometric information of each polygon, that is, the coordinate points of the outer contour of the polygon. Calculate its serial number from the inflection point coordinates of the polygon. The key code is as follows:

[0139]

[0140] 2) Read the pixel coordinate matrix file to obtain the physical coordinates corresponding to the inflection points of each polygon in the vector map patch. For calculating the planting area, only the planar area projected by the orthophoto image is required, so only the (x, y) coordinates are needed.

[0141]

[0142] 3) Calculate the area through the coordinates of each point of the polygon, summarize the polygon areas according to the crop planting types, and then the categories and planting areas of the crops around the measurement points can be calculated.

[0143]

[0144]

[0145] 4) Evaluation, verification and application. Due to the influence of various factors such as the atmospheric environment, instrument operation, data processing, etc., errors are inevitable. These errors may gradually accumulate and expand as the calculation range expands. To ensure the accuracy and reliability of the calculation results, strict evaluation and verification of the calculation results are required.

[0146] a. Analysis of error sources

[0147] Atmospheric environment: Components such as water vapor and aerosol in the atmosphere will affect the accuracy of the laser point cloud data, resulting in measurement errors.

[0148] Instrument operation: The calibration accuracy of the laser scanner and camera, the operation level of the operator, etc. will all affect the accuracy of the data.

[0149] Data processing: The algorithm selection and parameter settings in steps such as data preprocessing, image segmentation, and coordinate transformation will also affect the final results.

[0150] b. Error evaluation methods

[0151] Comparative analysis: Select multiple test areas with known actual areas, use the calculation method to obtain the areas of these areas, and compare them with the actual areas to calculate the errors.

[0152] Error statistics: Record the errors of each test area, calculate statistical quantities such as the average error and standard deviation, and evaluate the distribution of the errors.

[0153] Accuracy verification: Evaluate the stability and reliability of the results through multiple repeated measurements and calculations, and determine the accuracy of the calculation method.

[0154] c. The following table is an error comparison table, which records the error situations within different calculation ranges, where the surveyed area is the square of twice the length of the midpoint side.

[0155] No. Midpoint side length Surveyed area (square meters) Surveyed area (mu) Error percentage (%) 1 30 3600 5 3.31 2 50 10000 15 4.17 3 100 40000 60 5.02 4 200 160000 240 7.55 5 300 360000 540 9.83 6 400 640000 960 13.29 7 500 1000000 1500 18.91

[0156] Through the above evaluation and verification methods, the accuracy and reliability of the calculation results can be ensured. In practical applications, the maximum calculation range with an error within the acceptable range can be selected as the best recommended value for practical applications. For example, if a precision of 90% is required, the calculation range with the largest midpoint side length of 300 meters for surveying and mapping corresponding to an error percentage within 10% is selected.

[0157] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements on some or all of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.< / pcl::pointxyzi> < / syncpolicy> < / pcl::pointxyzi>

Claims

1. A method for identifying the planting distribution of crops based on deep learning and fusion positioning technology, characterized in that, It includes the following: Step 1: Obtain the camera image, 3D point cloud, and geographical coordinates of the target area through a collection device; Step 2: Identify the crops in the camera image through a deep learning algorithm, and at the same time identify the boundaries of the planting areas of each crop, and output the classified vector patches of the crops; Step 3: Use PCL to project the lidar point cloud data onto the camera image, fuse the camera image with the 3D point cloud, obtain the accurate coordinates of each pixel point in the camera image relative to the survey point, and output the pixel coordinate matrix; Step 4: Calculate the area of each crop distribution area based on the coordinates of the pixel points on the boundaries of each distribution area in the classified vector patches of the crops, summarize the same crops, and finally count the planting area and distribution of each crop.

2. The method for identifying the planting distribution of crops based on deep learning and fusion positioning technology according to claim 1, wherein The said Step 2 includes the following: Step 2.1: Select a binary-based classifier, regard each label as an independent binary classification problem, and achieve pixel-level multi-label classification through end-to-end training; Step 2.2: According to the phenological map, collect field photos of each region, each season, and each weather. For the collected image samples, perform image enhancement processing, as well as image cropping and mosaicking; Step 2.3: Label the samples. After labeling the images, the distribution areas of each crop and the negative list can be seen. The negative list includes wasteland, pond, greenhouse, orchard, vegetable plot, grassland; Step 2.4: The model configuration adopts SegNeXt based on the convolutional attention architecture scheme, and the multi-scale convolutional attention module MSCAN_L; Step 2.5: Model construction and prediction; Step 2.6: The metrics used for model evaluation and parameter tuning include mean average precision, precision, and recall. Repeatedly adjust the learning rate and batch size hyperparameters until the target training effect is achieved; Step 2.7: When the model prediction accuracy reaches the target training effect, the shp vector file output by the model is the final result, that is, the classified vector patches of the crop distribution, and the classified vector patches of the crop distribution indicate the distribution areas of each crop.

3. The method for identifying the planting distribution of crops based on deep learning and fusion positioning technology according to claim 2, wherein The image enhancement processing in the said Step 2.2 includes the following: For those with overly concentrated random distribution of pixel brightness and small contrast, adjust them to a normal distribution; grade the gray values of the gray image pixels and assign different colors; stretch the gray range of the image to the saturated state in the range of 0-255, and then swap the positive and negative pixels; perform noise removal, image smoothing, and sharpening operations on the pixels of each region according to a 5x5 filter.

4. The method for identifying the planting distribution of crops based on deep learning and fusion positioning technology according to claim 2, characterized in that, The image cropping and mosaicking in the said Step 2.2 includes the following: Image fusion; retain the areas of interest such as the crop planting area, and remove excessive sky and ground backgrounds; crop according to the longitude and latitude range of the geographical coordinates; centered on the target point, for multiple images taken around, after removing duplicates using the deep learning imagededup library, splice them into a whole to describe the planting situation of the current area.

5. The method for identifying the planting distribution of crops based on deep learning and fusion positioning technology according to claim 2, wherein The said Step 3 includes the following: Step 3.1: Use PCL to project the lidar point cloud data onto the camera image. The PCL is a function library for processing 3D point cloud data, including point cloud filtering, registration, segmentation, and feature extraction; Step 3.2: Georegister and project-correct the image and point cloud data to ensure data accuracy and consistency. Handle the conversion between radar coordinates and camera coordinates. To convert the radar coordinate system to the camera coordinate system, it is necessary to rotate counterclockwise by 90 degrees around the x-axis, then rotate clockwise by 90 degrees around the y-axis, and rotate 0 degrees around the z-axis. The rotation matrix is as follows: Step 3.3: Implement the above conversion from the radar coordinate system to the camera coordinate system using C++ code. Finally, project all the radar point cloud data onto the camera image. The algorithm is as follows: Define a structure that contains the topics of the camera and lidar, the camera internal parameter matrix camIn, and the RT matrix; Initialize the structure, read the topic names of the camera and lidar, the camera internal parameter matrix, and parameters such as the RT matrix from the configuration file, and store them in the corresponding variables of the structure iPara; Fuse the camera image and the radar point cloud data. First, create two message filters to subscribe to the camera image and the lidar point cloud data respectively, then use the ApproximateTime strategy for synchronization. Then create an image transport object and publish the processed image data. Finally, enter the message loop to project the received point cloud data onto the image for fusion; Step 3.4: Convert the image data in the message to the Mat format of OpenCV, and convert the lidar point cloud data to pcl::PointCloud <pcl::pointxyzi>Format, store it in cloud, then copy the original image data to visImg, and traverse the lidar point cloud data. For each point, perform the following processing:< / pcl::pointxyzi> If the x coordinate of the point is greater than maxX or less than 0, the absolute value of the y coordinate is greater than maxY, or the z coordinate is less than minZ, then skip this point; Store the x, y, z coordinates of the point and 1 into X; Calculate the fused point coordinates Y = camIn * RT * X; Map the Y coordinates to the image coordinate system, calculate the color according to the x value of the point, and finally draw a circle on the overlay image; After processing all the point cloud data, set the image encoding format to bgr8. Finally, publish the processed image data through image_publisher; Step 3.5: The data of each point in the obtained three-dimensional point cloud data structure pcl::PointXYZI corresponds to the three-dimensional coordinates of a pixel point in the image, that is, the pixel coordinate matrix. Traverse the point cloud data and save the data to a csv file to provide a basis for the next step of area calculation.

6. The method for identifying the planting distribution of crops based on deep learning and fusion positioning technology according to claim 5, characterized in that, The said Step 4 includes the following: Step 4.1: Read the crop distribution vector patch file to obtain the planting areas of various crops; use the geopandas library to read the previously stored shp file, extract the coordinate points of the outer contour of each polygon, and calculate its serial number from the inflection point coordinates of the polygon; Step 4.2: Read the pixel coordinate matrix file to obtain the physical coordinates corresponding to the inflection points of each polygon in the vector patch; Step 4.3: Calculate the area through the coordinates of each point of the polygon, summarize the polygon areas by crop planting type, and then the categories and planting areas of the crops around the measurement point can be calculated.

7. The method for identifying the planting distribution of crops based on deep learning and fusion positioning technology according to any one of claims 1-6, characterized in that, The acquisition device in the said Step 1 includes a satellite positioning device, a radar, and a camera.

Images