Crop planting information identification method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a crop planting information identification method and device, equipment and a storage medium, and is applied to the technical field of agricultural data processing. Preprocessing a to-be-identified crop image, and inputting the preprocessed to-be-identified crop image into the crop instance segmentation model to obtain a crop planting category and a crop planting area output by the model; the crop instance segmentation model is generated by constraint of a difficult case mining algorithm based on Fisher loss and K nearest neighbor, and the crop instance segmentation model is used for processing crop features in the to-be-identified crop image; and determining a crop planting area based on the crop planting area. Generation of a crop instance segmentation model is constrained through Fisher loss and a K-nearest neighbor-based difficult case mining algorithm, so that crop planting types and crop planting areas in crop images can be automatically identified in a complex scene, and the planting area is further calculated. The crop planting category identification precision and the crop planting area calculation precision are improved.

Description

Technical Field

[0001] The present application relates to the technical field of agricultural data processing, and specifically to a method, device, equipment and storage medium for identifying planting information of crops. Background Art

[0002] When it comes to individuals and small and micro enterprises in the agriculture and forestry industries who apply for loans from banks, banks mainly rely on manual on-site inspections to obtain agricultural production data such as crop types and planting areas. Factors such as weather conditions, topography, business personnel capabilities and performance have a significant impact on the collection of agricultural production data, making it difficult to directly, in real time, and truly control the actual production and operation conditions of loan individuals and loan enterprises, which makes it easy for risk events such as false mortgages to occur.

[0003] The existing technology uses drones and YOLOv3 models to identify crops. The drones are used to collect and organize image data of crop planting types, and the crop types in each image are labeled. Then, the YOLOv3 convolutional neural network is used to train the drone-based crop instance segmentation model, in which the feature extraction network uses Darknet-53, and the batch normalization layer (Batch Normalization, BN) is replaced by the group normalization layer (Group Normalization, GN) to reduce the impact of the batch size (Batch Size) on network training. Finally, the crop image to be identified is input into the trained model to obtain the recognition result.

[0004] The YOLOv3 model used in the prior art is a target detection model, and the output results are category labels and rectangular detection frames. However, in actual scenarios, the shape of farmland is irregular, and the rectangular detection frame cannot accurately represent the actual planting area of ​​the farmland, and the calculation of the crop planting area cannot be realized; and the detection frame is easily suppressed in densely overlapping scenarios, and the crop recognition accuracy is low. In addition, crop planting has characteristics such as spatial diversity, seasonal variability, and dynamics. The YOLOv3 model cannot be well applied to such detection scenarios, and it is difficult to extract discriminative classification features. It can be seen that the prior art has the problem of low recognition accuracy of crop types and planting areas. Summary of the invention

[0005] In view of this, the present application provides a method, device, equipment and storage medium for identifying crop planting information. By constraining the generation of a crop instance segmentation model through Fisher loss and a hard example mining algorithm based on K-nearest neighbors, it can automatically identify the crop planting types and crop planting areas in crop images in complex scenarios, and further calculate the planting area, thereby improving the accuracy of crop planting category recognition and crop planting area calculation.

[0006] To solve the above problems, the technical solutions provided by this application are as follows:

[0007] In a first aspect, an embodiment of the present application provides a method for identifying planting information of crops, wherein the planting information includes the planting category of crops and the planting area of ​​crops, and the identification method includes:

[0008] In response to collecting the image of the crop to be identified, preprocessing the image of the crop to be identified to obtain a preprocessed image of the crop to be identified;

[0009] Inputting the preprocessed crop image to be identified into a pre-constructed crop instance segmentation model to obtain the crop planting category and crop planting area output by the pre-constructed crop instance segmentation model; wherein the crop instance segmentation model is generated by a difficult example mining algorithm constraint based on Fisher loss and K nearest neighbors, and the crop instance segmentation model is used to process the crop features in the crop image to be identified;

[0010] The crop planting area is determined based on the crop planting region.

[0011] In a possible implementation, the training of the crop instance segmentation model includes:

[0012] Collecting a crop sample set, wherein the crop sample set includes crop images to be trained and crop labels, and the crop labels include the planting category and planting area of ​​the crop images to be trained;

[0013] Preprocessing the crop image to be trained to obtain a preprocessed crop image to be trained;

[0014] Inputting the crop image to be trained into an initial crop instance segmentation model to obtain training prediction data output by the initial crop instance segmentation model, wherein the initial crop instance segmentation model is used to process crop sample features in the crop image to be trained;

[0015] Based on the training prediction data and the crop labels, a classification loss, a regression loss, an instance segmentation loss, and a Fisher loss are calculated using a cross entropy loss function, an L1 smoothing loss function, a binary cross entropy loss function, and a Fisher loss function, respectively;

[0016] Updating the classification loss based on a hard example mining algorithm based on K nearest neighbors;

[0017] Determine a total loss based on the updated classification loss, the regression loss, the instance segmentation loss, and the Fisher loss;

[0018] Using the total loss to adjust parameters of the initial crop instance segmentation model;

[0019] If it is determined that the average precision of the initial crop instance segmentation model after the parameters are adjusted meets a preset condition, the initial crop instance segmentation model after the parameters are adjusted is used as the crop instance segmentation model.

[0020] In a possible implementation, preprocessing the to-be-identified crop image to obtain the preprocessed to-be-identified crop image includes:

[0021] De-noising, color correction and contrast enhancement are performed on the crop image to be identified to obtain an intermediate image of the crop; the crop image to be identified includes at least two aerial images of the crop;

[0022] Extracting key points from the crop intermediate image using a scale-invariant feature transform (SIFT) feature extraction operator, wherein the key points are used to calculate a local feature descriptor of the crop intermediate image;

[0023] Based on the local feature descriptor, determining matching points between adjacent crop intermediate images by a feature matching algorithm, wherein the matching points are used to determine relative positions between the adjacent crop intermediate images;

[0024] Based on the relative position, a weighted average of pixels in the overlapping area is calculated using a fade-in and fade-out weighted average fusion method;

[0025] According to the weighted average value of pixels in the overlapping area, a simulated annealing algorithm is used to optimize the image stitching parameters;

[0026] The preprocessed crop image to be identified is determined using the optimized image stitching parameters.

[0027] In a possible implementation, determining the crop planting area based on the crop planting region includes:

[0028] Determining the number of pixels in the crop planting area and the flight height of a collection device for collecting images of the crops to be identified;

[0029] The crop planting area is calculated based on the number of pixel points and the flight height of the acquisition device.

[0030] In a second aspect, an embodiment of the present application provides a device for identifying planting information of crops, wherein the planting information includes the planting category of crops and the planting area of ​​crops, and the identification device includes:

[0031] A preprocessing module, configured to preprocess the image of the crop to be identified in response to the acquisition of the image of the crop to be identified, so as to obtain the preprocessed image of the crop to be identified;

[0032] A recognition module, used for inputting the preprocessed crop image to be recognized into a pre-built crop instance segmentation model, and obtaining the crop planting category and crop planting area output by the pre-built crop instance segmentation model; wherein the crop instance segmentation model is generated by the constraints of the hard example mining algorithm based on Fisher loss and K nearest neighbors, and the crop instance segmentation model is used for processing the crop features in the crop image to be recognized;

[0033] A determination module is used to determine the crop planting area based on the crop planting area.

[0034] In a possible implementation, the recognition device further includes a model training module, and the model training module is used to train the crop instance segmentation model, including:

[0035] A collection submodule, used for collecting a crop sample set, wherein the crop sample set includes a crop image to be trained and a crop label, wherein the crop label includes a planting category and a planting area of ​​the crop image to be trained;

[0036] A preprocessing submodule, used for preprocessing the crop image to be trained to obtain a preprocessed crop image to be trained;

[0037] an identification submodule, used for inputting the crop image to be trained into an initial crop instance segmentation model to obtain training prediction data output by the initial crop instance segmentation model, wherein the initial crop instance segmentation model is used for processing crop sample features in the crop image to be trained;

[0038] A calculation submodule, for calculating classification loss, regression loss, instance segmentation loss and Fisher loss using a cross entropy loss function, an L1 smoothing loss function, a binary cross entropy loss function and a Fisher loss function respectively based on the training prediction data and the crop label;

[0039] An updating submodule, used for updating the classification loss based on a K-nearest neighbor hard example mining algorithm;

[0040] Determine a total loss based on the updated classification loss, the regression loss, the instance segmentation loss, and the Fisher loss;

[0041] A parameter adjustment submodule, used to adjust the parameters of the initial crop instance segmentation model using the total loss;

[0042] The determination submodule is used to determine that if the average accuracy of the initial crop instance segmentation model after parameter adjustment meets a preset condition, the initial crop instance segmentation model after parameter adjustment is used as the crop instance segmentation model.

[0043] In a possible implementation, the preprocessing module is used to preprocess the crop image to be identified to obtain the preprocessed crop image to be identified, including:

[0044] An acquisition submodule is used to remove noise, perform color correction and contrast enhancement on the crop image to be identified to obtain an intermediate image of the crop; the crop image to be identified includes at least two aerial images of the crop;

[0045] An extraction submodule, used for extracting key points from the crop intermediate image using a scale-invariant feature transform (SIFT) feature extraction operator, wherein the key points are used for calculating a local feature descriptor of the crop intermediate image;

[0046] A determination submodule, used to determine matching points between adjacent crop intermediate images through a feature matching algorithm based on the local feature descriptor, wherein the matching points are used to determine relative positions between the adjacent crop intermediate images;

[0047] A calculation submodule, configured to calculate a weighted average value of pixels in the overlapping area based on the relative position by using a fade-in and fade-out weighted average fusion method;

[0048] A stitching submodule, used for optimizing image stitching parameters by using a simulated annealing algorithm according to a weighted average value of pixels in the overlapping area;

[0049] The determination submodule is further used to determine the pre-processed crop image to be identified using the optimized image stitching parameters.

[0050] In a possible implementation, the determining module is specifically configured to:

[0051] Determine the number of pixels in the crop planting area and the flight height of a collection device that collects images of crops to be identified; and calculate the crop planting area based on the number of pixels and the flight height of the collection device.

[0052] In a third aspect, an embodiment of the present application provides a device for identifying planting information of crops, the device comprising: a processor, a memory, and a system bus;

[0053] The processor and the memory are connected via the system bus;

[0054] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor executes the method for identifying crop planting information described in the first aspect above.

[0055] In a fourth aspect, an embodiment of the present application provides a storage medium, wherein the storage medium stores instructions, and when the instructions are executed by a computer, the computer implements the method for identifying crop planting information described in the first aspect.

[0056] It can be seen that this application has the following beneficial effects:

[0057] The embodiment of the present application provides a method for identifying crop planting information, wherein the planting information includes crop planting categories and crop planting areas, and the identification method includes: first, in response to collecting a crop image to be identified, preprocessing the crop image to be identified to obtain a preprocessed crop image to be identified; second, inputting the preprocessed crop image to be identified into a pre-constructed crop instance segmentation model to obtain a crop planting category and a crop planting area output by the crop instance segmentation model; wherein the crop instance segmentation model is generated by a difficult example mining algorithm based on Fisher loss and K nearest neighbors, and the crop instance segmentation model is used to process crop features in the crop image to be identified; finally, determining the crop planting area based on the crop planting area. In this way, by constraining the generation of the crop instance segmentation model based on Fisher loss and a difficult example mining algorithm based on K nearest neighbors, the crop planting category and crop planting area in the crop image can be automatically identified in a complex scene, and the planting area is further calculated, thereby improving the accuracy of crop planting category recognition and crop planting area calculation.

[0058] The embodiment of the present application also provides a device corresponding to the above method, which has the same beneficial effects as the above method. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are some implementation methods of the present invention, and for ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0060] Figure 1 A schematic diagram of a crop instance segmentation model training and use process provided in an embodiment of the present application;

[0061] Figure 2 A schematic diagram of a method for identifying crop planting information provided in an embodiment of the present application;

[0062] Figure 3 A schematic diagram of a flow chart of a method for merging overlapping regions provided in an embodiment of the present application;

[0063] Figure 4A schematic diagram of a process for identifying crop planting categories and extracting planting areas provided in an embodiment of the present application;

[0064] Figure 5 A schematic diagram of a training process of a crop instance segmentation model provided in an embodiment of the present application;

[0065] Figure 6 A schematic diagram of the structure of a device for identifying crop planting information provided in an embodiment of the present application;

[0066] Figure 7 A schematic diagram of the structure of a crop planting information identification device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0068] In this application, the terms "comprises", "comprising" or any other variation thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0069] At present, it is difficult for banks to collect agricultural production data of individuals or enterprises that provide loans, resulting in a lack of objective and real-time data to cross-validate the risk of individual or enterprise loans. Based on drone technology, IoT devices equipped with cameras, positioning, sensors, etc. can collect crop planting information and report it to the bank's back-end system in real time; the back-end system can use deep learning, parallel computing and other technologies to analyze the information in real time to obtain agricultural planting data. There are many types of crops. There are significant differences in morphology, color, texture and other characteristics within the same type of crops. Even within the same variety, different characteristics will be presented due to various factors such as the growth environment and growth stage. At the same time, different types of crops will also have similarities in morphology, color, etc., especially in complex farmland environments, where different types of crops may be mixed together. The intra-class diversity and inter-class similarity problems in crop recognition bring huge challenges to the recognition of crop planting categories and crop planting areas.

[0070] In summary, in order to grasp agricultural production data in real time and improve the level of bank loan risk management, the embodiments of the present application provide a method, device, equipment and storage medium for identifying crop planting information. The planting information includes crop planting categories and crop planting areas. The identification method includes: first, in response to collecting crop images to be identified, preprocessing the crop images to be identified to obtain preprocessed crop images to be identified; secondly, inputting the preprocessed crop images to be identified into a pre-constructed crop instance segmentation model to obtain crop planting categories and crop planting areas output by the crop instance segmentation model; wherein the crop instance segmentation model is generated by constraints of a difficult example mining algorithm based on Fisher loss and K nearest neighbors, and the crop instance segmentation model is used to process crop features in the crop images to be identified; finally, determining the crop planting area based on the crop planting area. In this way, by constraining the generation of crop instance segmentation models through Fisher loss and K-nearest neighbor-based hard example mining algorithms, it is possible to automatically identify crop types and crop planting areas in crop images in complex scenarios, and further calculate the planting area, thereby improving the accuracy of crop planting category recognition and crop planting area calculation.

[0071] To facilitate understanding of the technical solution provided in the embodiment of the present application, a method, device, equipment and storage medium for identifying planting information of crops provided in the embodiment of the present application are described below in conjunction with the accompanying drawings.

[0072] First, the terms that appear in the embodiments of the present application are explained.

[0073] Instance segmentation is one of the basic tasks in the field of computer vision. It can not only identify the instance category in the image, but also generate a pixel-level segmentation mask for each instance to accurately mark the instance boundary. Instance segmentation usually includes four key steps: feature extraction, candidate region generation, classification and regression, and pixel-level segmentation.

[0074] Fisher discriminant analysis is a classic machine learning classification method. Its idea is to try to find a new feature space. When the training sample set is projected into the feature space, the distance between samples of the same type can be as small as possible and the distance between samples of different types can be as large as possible. When classifying new samples, they are projected into the same feature space, and then the category to which the new samples belong is determined based on the projection position. Fisher loss is a loss function designed for deep learning model training based on the idea of ​​Fisher discriminant analysis, which can help the model extract more discriminative features.

[0075] K-nearest neighbor learning is a classic lazy learning algorithm with no explicit training process. When classifying a test sample, K-nearest neighbor learning first finds the K training samples in the training set that are closest to the test sample, and then predicts the category of the test sample based on the categories of the K “neighbors”.

[0076] Hard example mining is an important training strategy in deep learning. It focuses on difficult samples in the model training process, that is, samples that are difficult for the model to correctly classify or predict.

[0077] See also Figure 1 , Figure 1 A schematic diagram of the training and use process of a crop instance segmentation model provided in an embodiment of the present application, wherein the crop instance segmentation model is trained in advance, and then the trained crop instance segmentation model is used to identify the planting type and extract the planting area of ​​the crop image, and then the planting area is calculated based on the planting area.

[0078] See also Figure 2 , Figure 2 A schematic flow chart of a method for identifying crop planting information provided in an embodiment of the present application. The method for identifying crop planting information can be applied to a server or a system, which is not limited in the embodiment of the present application and can be selected according to actual needs. The planting information includes the crop planting category and the crop planting area, and the identification method specifically includes S201-S203.

[0079] S201: In response to collecting an image of a crop to be identified, preprocessing the image of the crop to be identified to obtain a preprocessed image of the crop to be identified.

[0080] In the embodiment of the present application, the crop image to be identified is an image obtained by drone aerial photography. In order to calculate the crop planting area through aerial images, the drone's flight altitude is fixed at H, and an overlapping shooting strategy is adopted to ensure the integrity and continuity of the photographed farmland area. After the image acquisition is completed, the overlapping areas need to be merged.

[0081] See also Figure 3 , Figure 3 A schematic diagram of a flow chart of a method for merging overlapping regions provided in an embodiment of the present application.

[0082] In a possible implementation, the crop image to be identified is preprocessed to obtain the preprocessed crop image to be identified, which may specifically include image preprocessing, feature extraction and matching, image registration, overlapping area merging and global optimization.

[0083] Image preprocessing is to use the OpenCV vision library to preprocess each crop image to be identified, including de-noising, color correction and contrast enhancement, to obtain an intermediate image of the crop; thereby improving the quality and consistency of the image, wherein the crop image to be identified includes at least two aerial images of the crops.

[0084] Feature extraction and matching uses the scale-invariant feature transform (SIFT) feature extraction operator to extract key points from each crop intermediate image, and the key points are used to calculate the local feature descriptor of the crop intermediate image.

[0085] Image registration is based on local feature descriptors. The matching points between adjacent crop intermediate images are determined through a feature matching algorithm. The matching points are used to achieve image registration. The relative positions between adjacent crop intermediate images are determined based on the matching points of the images to achieve image alignment.

[0086] The overlapping areas are merged based on relative positions, and the weighted average fusion method with fade-in and fade-out is used to calculate the weighted average of the pixels in the overlapping area, and it is used as the final pixel value of the overlapping area.

[0087] The global optimization is to optimize the image stitching parameters according to the weighted average value of pixels in the overlapping area using the simulated annealing algorithm to reduce the stitching gap and color difference so that the merged image has a natural transition in the overlapping area.

[0088] Finally, the crop intermediate images are stitched using the optimized image stitching parameters to determine the preprocessed crop image to be identified.

[0089] After the aforementioned preprocessing, the crop images to be identified have enhanced features related to the identification of crop planting types and planting areas, which helps to improve the recognition accuracy and efficiency.

[0090] S202: Input the preprocessed crop image to be identified into a pre-constructed crop instance segmentation model to obtain the crop planting category and crop planting area output by the pre-constructed crop instance segmentation model; wherein the crop instance segmentation model is generated based on the constraints of the difficult example mining algorithm based on Fisher loss and K nearest neighbors, and the crop instance segmentation model is used to process the crop features in the crop image to be identified.

[0091] The crop instance segmentation model is obtained through training. After the model training is completed, the weight file with the highest mean average precision (mAP) value can be selected as the prediction model parameter, and then the crop type recognition and planting area extraction can be performed on the crop images to be identified taken by the drone.

[0092] For the specific process of training the crop instance segmentation model provided in the embodiment of the present application, please see below.

[0093] See also Figure 4 , Figure 4 A schematic diagram of a process for identifying crop planting categories and extracting planting areas provided in an embodiment of the present application.

[0094] For the recognition of crop planting categories, the softmax function in the crop instance segmentation model is used to convert the original score of each planting category in each crop image to be identified into a probability value [0,1] for each planting category, and obtain an N*C probability matrix. Wherein, N is the number of candidate areas, and the value of N can be 1000. The embodiment of the present application is not limited to this, and the value of N can be adjusted according to actual needs. C is the number of crop planting categories in the crop image to be identified. By comparing the probability values ​​of each planting category, the crop planting category that each candidate area is most likely to contain can be determined.

[0095] For the extraction of crop planting areas, the crop instance segmentation model processes the crop features in the crop image to be identified, where the crop feature vector is passed through the regression head and the mask head to obtain the coordinate offset N*4 of the predicted bounding box and the three-dimensional feature map N*W*H of the same size as the candidate box. The coordinate offset usually includes the offset of the center point (Δx, Δy) and the scaling factors of the width and height (Δw, Δh).

[0096] Assuming that the center coordinates of the original candidate box are (x_c, y_c), the width and height are (w, h), the center coordinates (x_p, y_p) and the width and height (w_p, h_p) of the predicted box can be expressed as:

[0097] x_p=x_c+Δx*w;

[0098] y_p=y_c+Δy*h;

[0099] w_p=w+Δw;

[0100] h_p=h+Δh.

[0101] Each value in the three-dimensional feature map N*W*H represents the confidence that the corresponding pixel belongs to the target object. In order to convert these continuous values ​​into a binary mask, a threshold processing method can be used. Among them, the values ​​in the three-dimensional feature map are usually floating point numbers. The threshold processing method is to set a threshold value η, set all pixel values ​​greater than η to 1, indicating that the pixel belongs to the target object; set all pixel values ​​less than η to 0, indicating that the pixel belongs to the background. The binary mask can accurately represent the position and shape of the target object, thereby realizing the extraction of the crop planting area.

[0102] The crop planting area extracted above needs to undergo non-maximum suppression, dilation and corrosion processing before being output to the next step for use.

[0103] Non-maximum suppression: The number of candidate regions N is often greater than the actual number of targets in crop images. In fact, there are often many candidate boxes around a target. In order to eliminate overlapping candidate boxes and select the best candidate box, the candidate boxes can be sorted according to the classification score. Starting from the box with the highest score, suppress all other boxes that overlap with it by more than a certain area, and iterate the above process repeatedly until no more boxes need to be suppressed. The intersection over union (IoU) represents the ratio of the intersection area of ​​the predicted box and the true box to the union area, which can be used in the above process to measure the degree of overlap between the candidate box and the true box. The calculation of the intersection over union (IoU) can be found in the following formula (1).

[0104]

[0105] Dilation and erosion: In order to improve the smoothness of the mask boundary and eliminate small noise or isolated areas, the embodiment of the present application first uses an erosion operation to remove noise and isolated areas, and then uses a dilation operation to fill small gaps and holes.

[0106] After the above processing, the output crop planting area conforms to the characteristics of the actual planting area, avoiding the recognition interference caused by overlapping areas.

[0107] S203: Determine the crop planting area based on the crop planting region.

[0108] After the above steps, we can get the binary mask of the crop planting area. A pixel value of 1 indicates that the pixel belongs to the target object, and a pixel value of 0 indicates that the pixel belongs to the background. Therefore, we only need to use numpy.sum() on the mask to count the number of pixels C in the mask area.

[0109] Calculation of the mapping relationship between pixel points and actual distance: The ground sampling distance (GSD) of the drone can reflect the relationship between the image pixel points and the actual distance. See the following formula (2):

[0110]

[0111] Among them, H is the flight height of the drone, f is the focal length of the camera lens on the drone, and a is the pixel size (in mm). The two parameters of f and a can be found in the information corresponding to the drone model, which can be found on the official website or instructions of the drone. When the drone shooting height H is fixed, the actual area represented by a pixel in the crop image can be calculated.

[0112] In one possible implementation, determining the crop planting area based on the crop planting region includes: determining the number of pixels in the crop planting region and the flight height of a collection device that collects images of crops to be identified; and calculating the crop planting area based on the number of pixels and the flight height of the collection device.

[0113] The crop planting area S=C*H.

[0114] In response to the problem that it is time-consuming and laborious to obtain real agricultural production data in the process of bank agricultural lending, the embodiment of the present application is based on the crop planting category recognition and planting area calculation method based on drone aerial images and deep learning, which can automatically count the crop planting types and corresponding planting areas, and provide timely, objective and real data sources for bank credit access, due diligence, post-loan monitoring and early warning, thereby improving the level of credit information management.

[0115] Based on the contents of the above steps S201-S203, it can be known that by applying Fisher loss to the learnable features of the crop instance segmentation model to constrain the learning process of the crop features, the crop features have a smaller intra-class distance and a larger inter-class interval, thereby improving the discrimination of the crop features. For crop features that are not clearly distinguished from other categories, K-nearest neighbor learning is used to convert the distance distribution between the crop features and the neighboring crop features into a K-nearest neighbor probability distribution, and the K-nearest neighbor probability is used as an indication of difficult and easy samples to guide the model to pay more attention to crops that are not easy to distinguish, thereby improving the recognition effect of such crops. The aerial crop image is input into the trained crop instance segmentation model to identify the crop type and extract the crop planting area, and finally the crop planting area is calculated in combination with the actual physical size corresponding to the single pixel. In this way, the crop planting type and crop planting area in the crop image can be automatically identified in complex scenes, and the planting area is further calculated, thereby improving the accuracy of crop planting category recognition and crop planting area calculation.

[0116] The specific process of training the crop instance segmentation model provided in the embodiment of the present application is described below.

[0117] See also Figure 5 , Figure 5 A schematic diagram of the training process of a crop instance segmentation model provided in an embodiment of the present application.

[0118] The collected crop sample set is subjected to feature extraction, region proposal box generation, pooling, and full connection layer to obtain predicted planting category, predicted planting area, coordinate offset, and mask information. Then, the classification loss is calculated using the cross entropy loss function, L1 smoothing loss function, and binary cross entropy loss function, and the classification loss is calculated using the Fisher loss function.cls , regression loss l reg , instance segmentation loss l mask and Fisher loss l fisher ; Finally, the model can be learned by calculating the gradient and back propagating the parameters, and the above process is repeated until the model reaches the specified accuracy or training rounds.

[0119] In order to improve the recognition accuracy of crops and the segmentation accuracy of planting areas, this patent adjusts the model loss function through Fisher loss and K-nearest neighbor-based difficult example mining to guide the learning process of the model and ultimately obtain discriminative crop features.

[0120] In one possible implementation, the training of a crop instance segmentation model includes:

[0121] A crop sample set is collected, the crop sample set includes crop images to be trained and crop labels, the crop labels include the planting category and planting area of ​​the crop images to be trained; the crop images to be trained are preprocessed to obtain the preprocessed crop images to be trained.

[0122] For preprocessing of the training crop images, please refer to the above content on processing of the crop images to be identified.

[0123] The crop image to be trained is input into the initial crop instance segmentation model to obtain training prediction data output by the initial crop instance segmentation model. The initial crop instance segmentation model is used to process the crop sample features in the crop image to be trained.

[0124] Based on the training prediction data and crop labels, the classification loss, regression loss, instance segmentation loss and Fisher loss are calculated using the cross entropy loss function, L1 smoothing loss function, binary cross entropy loss function and Fisher loss function respectively; the classification loss is updated based on the K-nearest neighbor hard example mining algorithm; the total loss is determined based on the updated classification loss, regression loss, instance segmentation loss and Fisher loss.

[0125] The total loss is used to adjust the parameters of the initial crop instance segmentation model; if the average accuracy of the initial crop instance segmentation model after the parameter adjustment meets the preset conditions, the initial crop instance segmentation model after the parameter adjustment is used as the crop instance segmentation model.

[0126] The Fisher loss function and K-nearest neighbor hard example mining algorithm are explained in detail.

[0127] Fisher loss function:

[0128] Assumptions represents a one-dimensional crop feature vector, where c represents the crop planting category, represents the feature set of the i-th type of crops. Based on the idea of ​​Fisher discriminant analysis, the Fisher loss function l fisher By minimizing χ FC The intra-class divergence S w

[0129] and maximize χ FC The inter-class divergence S b Implementation. Among them, S w and S b The definitions are given in the following formulas (3) and (4):

[0130]

[0131] Where C represents the number of crop planting categories, n i represents the number of samples of the i-th type of crops, m i and m are and χ FC The characteristic mean of a certain type of crop, that is, the intra-class characteristic mean and the overall characteristic mean, is calculated as follows:

[0132]

[0133] Where n represents χ FC The number of crops in , that is, the total number of crops.

[0134] Finally, define the Fisher loss function l fisher for:

[0135]

[0136] Here, tr(·) represents the trace of the matrix.

[0137] According to the back propagation training method of deep neural network, Fisher loss function l fisher The process of continuous optimization and reduction is the process of updating crop features in the direction of small intra-class spacing and large inter-class spacing. The classification, regression, and segmentation tasks of the final model will use the optimized crop features for crop identification and precise positioning.

[0138] K-nearest neighbor hard example mining algorithm:

[0139] In order to guide the model to pay more attention to crops that are difficult to identify during the training stage, the embodiment of the present application proposes using K-nearest neighbor learning to convert the distance distribution between crop features into a K-nearest neighbor probability distribution, and then use the K-nearest neighbor probability as an indication of difficult and easy samples.

[0140] Step a: Find K nearest neighbors.

[0141] Calculate χ using the following formula (8): FC The Euclidean distance between any two crop features in

[0142]

[0143] Where d represents χ FC The dimension of the feature vector in (generally, d = 1024); for each crop feature χ, select K features that are closest to it except itself, denoted as Represents a collection of neighbor features The feature set of crops belonging to planting category c in .

[0144] Step b: Calculate the similarity.

[0145] The greater the distance between two feature vectors, the lower the similarity should be; the smaller the distance between two feature vectors, the higher the similarity should be. In order to achieve the conversion between feature distance and feature similarity, it is necessary to define a similarity calculation function that meets at least the following two requirements: first, ensure that the similarity between any two feature vectors is greater than 0; second, ensure that the similarity is inversely proportional to the distance between the feature vectors. Based on this, the embodiment of the present application defines a similarity calculation function as shown in formula (9).

[0146]

[0147] In addition to meeting the above two basic requirements, defining the similarity calculation function in the form of exponential decay can also amplify the differences between neighbors. As long as the distance between neighbors increases, the similarity decreases rapidly. This helps reduce the impact of distant neighbors on subsequent probability calculations.

[0148] Step c: Calculate the probability distribution.

[0149] Convert the distance between the crop feature χ and its K nearest neighbors into a probability distribution over C planting categories. knn (c|χ) represents the probability that the sample feature χ is predicted to be class c.

[0150]

[0151] Step d: Hard example mining.

[0152] p_knn represents the probability that the sample feature is predicted as the true category, and defines the adjustment factor w = (1-p knn ) 2 , which is used to adjust the contribution of easy-to-identify crops and difficult-to-identify crops to the classification loss, thereby updating the classification loss. The final classification loss is lcls_knn =(1+w)l cls .

[0153] The total loss in the final model training phase is l = (1 + w) l cls +l reg +l fisher +l mask .

[0154] Factors such as time, space, variety, and human management can lead to differences in appearance between crops of the same type, while crops of different types are similar in appearance and morphology, which greatly increases the difficulty of crop recognition by neural networks. In view of the intra-class diversity and inter-class similarity problems in crop recognition scenarios, the embodiment of the present application applies Fisher constraints on sample features to guide the learning process of sample features through a Fisher loss function based on feature constraints, and finally obtains discriminative classification features with small intra-class intervals and large inter-class distances, thereby improving the accuracy of crop recognition and segmentation.

[0155] For crops that are difficult to distinguish, a difficult example mining algorithm based on K-nearest neighbors is used. K-nearest neighbor learning is used to process crop features to obtain K-nearest neighbor probabilities. The K-nearest neighbor probabilities are used as indicators of easy-to-distinguish and difficult-to-distinguish crops, increasing the loss ratio of difficult example samples to guide the neural network to pay more attention to crops that are difficult to distinguish in the future, thereby improving the model's ability to extract features for difficult-to-identify crops and improving crop recognition accuracy.

[0156] The above-mentioned embodiment of the present application provides a method for identifying crop planting information based on the above-mentioned method. Next, a device for identifying crop planting information provided in the embodiment of the present application is described, which is used to perform the above-mentioned Figure 2 The function of the crop planting information identification device is described. The structural diagram of the crop planting information identification device is shown in FIG. Figure 6 As shown, it includes a pre-processing module 601 , an identification module 602 and a determination module 603 .

[0157] Among them, the planting information includes crop planting categories and crop planting areas.

[0158] A preprocessing module 601 is used for preprocessing the crop image to be identified in response to collecting the crop image to be identified, so as to obtain the preprocessed crop image to be identified;

[0159] The recognition module 602 is used to input the preprocessed crop image to be recognized into a pre-built crop instance segmentation model to obtain the crop planting category and crop planting area output by the pre-built crop instance segmentation model; wherein the crop instance segmentation model is generated by the constraints of the hard example mining algorithm based on Fisher loss and K nearest neighbors, and the crop instance segmentation model is used to process the crop features in the crop image to be recognized;

[0160] The determination module 603 is used to determine the crop planting area based on the crop planting area.

[0161] In a possible implementation, the recognition device further includes a model training module, and the model training module is used to train the crop instance segmentation model, including:

[0162] A collection submodule, used for collecting a crop sample set, wherein the crop sample set includes a crop image to be trained and a crop label, wherein the crop label includes a planting category and a planting area of ​​the crop image to be trained;

[0163] A preprocessing submodule, used for preprocessing the crop image to be trained to obtain a preprocessed crop image to be trained;

[0164] an identification submodule, used for inputting the crop image to be trained into an initial crop instance segmentation model to obtain training prediction data output by the initial crop instance segmentation model, wherein the initial crop instance segmentation model is used for processing crop sample features in the crop image to be trained;

[0165] A calculation submodule, for calculating classification loss, regression loss, instance segmentation loss and Fisher loss using a cross entropy loss function, an L1 smoothing loss function, a binary cross entropy loss function and a Fisher loss function respectively based on the training prediction data and the crop label;

[0166] An updating submodule, used for updating the classification loss based on a K-nearest neighbor hard example mining algorithm;

[0167] Determine a total loss based on the updated classification loss, the regression loss, the instance segmentation loss, and the Fisher loss;

[0168] A parameter adjustment submodule, used to adjust the parameters of the initial crop instance segmentation model using the total loss;

[0169] The determination submodule is used to determine that if the average accuracy of the initial crop instance segmentation model after parameter adjustment meets a preset condition, the initial crop instance segmentation model after parameter adjustment is used as the crop instance segmentation model.

[0170] In a possible implementation, the preprocessing module 601 is used to preprocess the crop image to be identified to obtain the preprocessed crop image to be identified, including:

[0171] An acquisition submodule is used to remove noise, perform color correction and contrast enhancement on the crop image to be identified to obtain an intermediate image of the crop; the crop image to be identified includes at least two aerial images of the crop;

[0172] An extraction submodule, used for extracting key points from the crop intermediate image using a scale-invariant feature transform (SIFT) feature extraction operator, wherein the key points are used for calculating a local feature descriptor of the crop intermediate image;

[0173] A determination submodule, used to determine matching points between adjacent crop intermediate images through a feature matching algorithm based on the local feature descriptor, wherein the matching points are used to determine relative positions between the adjacent crop intermediate images;

[0174] A calculation submodule, configured to calculate a weighted average value of pixels in the overlapping area based on the relative position by using a fade-in and fade-out weighted average fusion method;

[0175] A stitching submodule, used for optimizing image stitching parameters by using a simulated annealing algorithm according to a weighted average value of pixels in the overlapping area;

[0176] The determination submodule is further used to determine the pre-processed crop image to be identified using the optimized image stitching parameters.

[0177] In a possible implementation, the determining module 603 is specifically configured to:

[0178] Determine the number of pixels in the crop planting area and the flight height of a collection device that collects images of crops to be identified; and calculate the crop planting area based on the number of pixels and the flight height of the collection device.

[0179] The embodiment of the present application provides a device for identifying crop planting information, including a preprocessing module, an identification module and a determination module. The planting information includes crop planting categories and crop planting areas. The preprocessing module is used to preprocess the crop image to be identified in response to collecting the crop image to be identified, so as to obtain the preprocessed crop image to be identified. The identification module is used to input the preprocessed crop image to be identified into a pre-constructed crop instance segmentation model to obtain the crop planting category and crop planting area output by the pre-constructed crop instance segmentation model; wherein the crop instance segmentation model is generated by the constraints of the difficult example mining algorithm based on Fisher loss and K nearest neighbors, and the crop instance segmentation model is used to process the crop features in the crop image to be identified. The determination module is used to determine the crop planting area based on the crop planting area. In this way, by constraining the generation of crop instance segmentation models through Fisher loss and K-nearest neighbor-based hard example mining algorithms, it is possible to automatically identify crop types and crop planting areas in crop images in complex scenarios, and further calculate the planting area, thereby improving the accuracy of crop planting category recognition and crop planting area calculation.

[0180] Based on the method for identifying crop planting information provided by the above method embodiment, the present application embodiment also provides a crop planting information identification device, see Figure 7 , the device includes: a processor, a memory, and a system bus;

[0181] The processor and the memory are connected via the system bus;

[0182] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor executes the method for identifying crop planting information described in any one of the above embodiments.

[0183] Based on the method for identifying the planting information of crops provided in the above method embodiment, the embodiment of the present application also provides a storage medium, which stores instructions. When the instructions are executed by a computer, the computer implements the method for identifying the planting information of crops described in any of the above embodiments.

[0184] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device or device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device and device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0185] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0186] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for identifying crop planting information, characterized in that: The planting information includes the crop planting category and the crop planting area, and the identification method includes: In response to collecting the image of the crop to be identified, preprocessing the image of the crop to be identified to obtain a preprocessed image of the crop to be identified; Inputting the preprocessed crop image to be identified into a pre-constructed crop instance segmentation model to obtain the crop planting category and crop planting area output by the pre-constructed crop instance segmentation model; wherein the crop instance segmentation model is generated by a difficult example mining algorithm constraint based on Fisher loss and K nearest neighbors, and the crop instance segmentation model is used to process the crop features in the crop image to be identified; The crop planting area is determined based on the crop planting region.

2. The identification method according to claim 1, characterized in that: The training of the crop instance segmentation model includes: Collecting a crop sample set, wherein the crop sample set includes crop images to be trained and crop labels, and the crop labels include the planting category and planting area of ​​the crop images to be trained; Preprocessing the crop image to be trained to obtain a preprocessed crop image to be trained; Inputting the crop image to be trained into an initial crop instance segmentation model to obtain training prediction data output by the initial crop instance segmentation model, wherein the initial crop instance segmentation model is used to process crop sample features in the crop image to be trained; Based on the training prediction data and the crop labels, a classification loss, a regression loss, an instance segmentation loss, and a Fisher loss are calculated using a cross entropy loss function, an L1 smoothing loss function, a binary cross entropy loss function, and a Fisher loss function, respectively; Updating the classification loss based on a hard example mining algorithm based on K nearest neighbors; Determine a total loss based on the updated classification loss, the regression loss, the instance segmentation loss, and the Fisher loss; Using the total loss to adjust parameters of the initial crop instance segmentation model; If it is determined that the average precision of the initial crop instance segmentation model after the parameters are adjusted meets a preset condition, the initial crop instance segmentation model after the parameters are adjusted is used as the crop instance segmentation model.

3. The identification method according to claim 1, characterized in that: The preprocessing of the crop image to be identified to obtain the preprocessed crop image to be identified includes: De-noising, color correction and contrast enhancement are performed on the crop image to be identified to obtain an intermediate image of the crop; the crop image to be identified includes at least two aerial images of the crop; Extracting key points from the crop intermediate image using a scale-invariant feature transform (SIFT) feature extraction operator, wherein the key points are used to calculate a local feature descriptor of the crop intermediate image; Based on the local feature descriptor, determining matching points between adjacent crop intermediate images by a feature matching algorithm, wherein the matching points are used to determine relative positions between the adjacent crop intermediate images; Based on the relative position, a weighted average of pixels in the overlapping area is calculated using a fade-in and fade-out weighted average fusion method; According to the weighted average value of pixels in the overlapping area, a simulated annealing algorithm is used to optimize the image stitching parameters; The preprocessed crop image to be identified is determined using the optimized image stitching parameters.

4. The identification method according to claim 1, characterized in that: The determining the crop planting area based on the crop planting area comprises: Determining the number of pixels in the crop planting area and the flight height of a collection device for collecting images of the crops to be identified; The crop planting area is calculated based on the number of pixel points and the flight height of the acquisition device.

5. A device for identifying crop planting information, characterized in that: The planting information includes the crop planting category and the crop planting area, and the identification device includes: A preprocessing module, configured to preprocess the image of the crop to be identified in response to the acquisition of the image of the crop to be identified, so as to obtain the preprocessed image of the crop to be identified; A recognition module, used for inputting the preprocessed crop image to be recognized into a pre-built crop instance segmentation model, and obtaining the crop planting category and crop planting area output by the pre-built crop instance segmentation model; wherein the crop instance segmentation model is generated by the constraints of the hard example mining algorithm based on Fisher loss and K nearest neighbors, and the crop instance segmentation model is used for processing the crop features in the crop image to be recognized; A determination module is used to determine the crop planting area based on the crop planting area.

6. The identification device according to claim 5, characterized in that: The recognition device further includes a model training module, which is used to train the crop instance segmentation model, including: A collection submodule, used for collecting a crop sample set, wherein the crop sample set includes a crop image to be trained and a crop label, wherein the crop label includes a planting category and a planting area of ​​the crop image to be trained; A preprocessing submodule, used for preprocessing the crop image to be trained to obtain a preprocessed crop image to be trained; an identification submodule, used for inputting the crop image to be trained into an initial crop instance segmentation model to obtain training prediction data output by the initial crop instance segmentation model, wherein the initial crop instance segmentation model is used for processing crop sample features in the crop image to be trained; A calculation submodule, for calculating classification loss, regression loss, instance segmentation loss and Fisher loss using a cross entropy loss function, an L1 smoothing loss function, a binary cross entropy loss function and a Fisher loss function respectively based on the training prediction data and the crop label; An updating submodule, used for updating the classification loss based on a K-nearest neighbor hard example mining algorithm; Determine a total loss based on the updated classification loss, the regression loss, the instance segmentation loss, and the Fisher loss; A parameter adjustment submodule, used to adjust the parameters of the initial crop instance segmentation model using the total loss; The determination submodule is used to determine that if the average accuracy of the initial crop instance segmentation model after parameter adjustment meets a preset condition, the initial crop instance segmentation model after parameter adjustment is used as the crop instance segmentation model.

7. The identification device according to claim 5, characterized in that: The preprocessing module is used to preprocess the crop image to be identified to obtain the preprocessed crop image to be identified, including: An acquisition submodule is used to remove noise, perform color correction and contrast enhancement on the crop image to be identified to obtain an intermediate image of the crop; the crop image to be identified includes at least two aerial images of the crop; An extraction submodule, used for extracting key points from the crop intermediate image using a scale-invariant feature transform (SIFT) feature extraction operator, wherein the key points are used for calculating a local feature descriptor of the crop intermediate image; A determination submodule, used to determine matching points between adjacent crop intermediate images through a feature matching algorithm based on the local feature descriptor, wherein the matching points are used to determine relative positions between the adjacent crop intermediate images; A calculation submodule, configured to calculate a weighted average value of pixels in the overlapping area based on the relative position by using a fade-in and fade-out weighted average fusion method; A stitching submodule, used for optimizing image stitching parameters by using a simulated annealing algorithm according to a weighted average value of pixels in the overlapping area; The determination submodule is further used to determine the pre-processed crop image to be identified using the optimized image stitching parameters.

8. The identification device according to claim 5, characterized in that: The determining module is specifically used for: Determine the number of pixels in the crop planting area and the flight height of a collection device that collects images of crops to be identified; and calculate the crop planting area based on the number of pixels and the flight height of the collection device.

9. A crop planting information identification device, characterized in that: The device includes: a processor, a memory, and a system bus; The processor and the memory are connected via the system bus; The memory is used to store one or more programs, and the one or more programs include instructions, which, when executed by the processor, enable the processor to execute the crop planting information identification method according to any one of claims 1 to 4.

10. A storage medium, characterized in that: The storage medium stores instructions, and when the instructions are executed by a computer, the computer implements the crop planting information identification method as described in any one of claims 1 to 4.

Images