Crop multi-granularity recognition method under multi-scale view

Abstract

The application discloses a multi-fine-granularity recognition method for crops under a multi-scale field of view, and belongs to the technical field of intelligent agricultural target detection and recognition, and comprises the following steps: acquiring infrared images of different scales and visible light images of different scales of crops in farmlands; pre-processing the infrared images and the visible light images; performing label annotation on the visible light images of different scales; performing registration on the visible light images of different scales, and then performing registration on the registered visible light images and the infrared images; fusing the registered visible light images and the infrared images to obtain a fusion feature map; inputting the fusion feature map and the visible light images with labels into a multi-fine-granularity recognition network for classification and recognition, and outputting the types and growth periods of the crops. The application has the advantages of small required data volume, high model efficiency, accurate recognition of the types and growth periods of the crops, and provision of a light-weight basis for precision agriculture.

Description

Technical Field

[0001] This invention belongs to the technical field of target detection and recognition in smart agriculture, specifically relating to a method for multi-scale, fine-grained identification of crops under multi-scale field of view. Background Technology

[0002] In the current transformation of smart agriculture towards precision and digitalization, crop identification, as a core preliminary step in farmland monitoring, growth assessment, pest and disease early warning, and precision fertilization management, directly determines the scientific nature of agricultural production decisions based on its accuracy and efficiency. Existing crop detection and identification methods largely rely on data from a "single field of view," such as only drone canopy images, failing to consider both macroscopic growth and microscopic features. Traditional identification methods often focus on a single dimension, making it difficult to meet fine-grained requirements. Due to the lack of fine-grained classification, they cannot provide accurate information for agricultural production. For example, if the crop growth stage cannot be accurately determined, targeted agricultural decisions become difficult, impacting crop yield and quality.

[0003] Currently, the main existing technical solutions in the field of crop detection and classification are as follows:

[0004] (1) Single sensor data detection: relying solely on visible light cameras to acquire crop images for detection and classification. This method provides limited data and makes it difficult to obtain comprehensive information about the crops.

[0005] (2) Simple integration of scale data processing: Although attempts were made to utilize multi-scale data, there was a lack of effective methods in processing the dataset, resulting in low detection accuracy.

[0006] (3) Coarse classification detection: Most existing crop detection methods are limited to coarse classification, such as distinguishing only broad categories like corn and soybeans, without identifying crop varieties and growth stages. These existing technologies have obvious technical shortcomings. For example, in a multi-scale perspective, the processing of datasets is difficult, making it hard to fully extract effective information from the data, which makes it difficult to improve detection accuracy. Therefore, it is necessary to design more effective multi-fine-grained crop identification methods to promote the development of smart agriculture. Summary of the Invention

[0007] The purpose of this invention is to address the aforementioned shortcomings in the prior art by providing a multi-scale field-of-view method for crop multi-fine-grained identification, thereby solving the problems of existing crop detection methods, such as difficulty in obtaining comprehensive crop information and low detection accuracy.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] A method for identifying crops at multiple scales and finer granularities, comprising the following steps:

[0010] S1. Acquire infrared images and visible light images of farmland crops at different scales;

[0011] S2, Preprocessing infrared and visible light images;

[0012] S3. Label and annotate visible light images at different scales;

[0013] S4. Register visible images at different scales, and then register the registered visible light images with infrared images;

[0014] S5. The registered visible light image and infrared image are fused to obtain a fused feature map;

[0015] S6. Input the fused feature map and the labeled visible light image into a multi-fine-grained recognition network for classification and recognition, and output the crop type and growth stage.

[0016] Furthermore, in S1, infrared images of different scales include infrared large receptive field images and infrared small receptive field images; visible light images of different scales include visible light large receptive field images and visible light small receptive field images.

[0017] Furthermore, in step S3, the visible light images at different scales are labeled, including:

[0018] Manual labeling was used to label species in visible light images with large receptive fields, and to label species and growth stage in visible light images with small receptive fields.

[0019] Furthermore, in S4, the homography registration method of ORB features is used to register the visible light large receptive field image and the visible light small receptive field image, and then the same ORB feature homography registration method is used to register the registered visible light small receptive field image with the infrared small receptive field image.

[0020] The homography registration method using ORB features is used to register visible light large receptive field images and visible light small receptive field images. This process includes the following steps:

[0021] S41. Hamming distance is used to measure the similarity of ORB feature point descriptors in order to match feature points;

[0022] The matching conditions for feature points are as follows:

[0023]

[0024] In the formula, The Hamming distance is the nearest neighbor. The Hamming distance to the next nearest neighbor; Indicates the threshold;

[0025] S42. Based on the homography matrix, the pixels of the visible light small receptive field image are mapped to the corresponding positions in the visible light large receptive field image. The coordinate transformation is expressed as follows:

[0026]

[0027] In the formula, These are the homogeneous coordinates of a visible light small receptive field image; The homogeneous coordinates of the visible light large receptive field image; It is a homography matrix.

[0028] Furthermore, in step S42, the RANSAC algorithm is used to filter interior points from feature point pairs containing multiple exterior points. Then, based on the filtered interior points, the least squares method is used to solve for the optimal homography matrix. This specifically includes the following sub-steps:

[0029] S421. Randomly select 4 pairs of non-collinear points from the initial matched point pairs. , , , , These represent the first, second, third, and fourth randomly selected feature point pairs, respectively.

[0030] S422, Based on 4 pairs of non-collinear points A system of linear equations for four pairs of points is constructed using the least squares method, and the temporary homography matrix is ​​solved:

[0031]

[0032] In the formula, This is a temporary homography matrix; The parameter represents the parameter that makes the function reach its minimum value; Indicates the first randomly selected... i For feature points; , Indicates the first i Homogeneous coordinates of the point; Euclidean distance;

[0033] S423. Calculate the reprojection error of the matching point pair based on the temporary homography matrix:

[0034]

[0035] In the formula, This is due to reprojection error; , represents the coordinates of feature points in the large receptive field image and the small receptive field image of visible light, respectively; w is the third dimension of the homogeneous coordinates of the feature points;

[0036] If reprojection error <Error threshold Then, the corresponding points are marked as interior points, and the set of interior points is denoted as . ;

[0037] S424. Repeat S421~S423 several times, retain the set of interior points with the largest number of interior points, and denote it as the optimal set of interior points. And at the same time retain the corresponding temporary homography matrix;

[0038] S425. For the optimal set of interior points The homogeneous coordinate transformation of the interior points satisfies:

[0039]

[0040] After expanding the homogeneous coordinates of the interior point, it is transformed into a system of linear equations:

[0041]

[0042] Solve this system of linear equations using the least squares method:

[0043]

[0044] In the formula, Scale factor; This is the coefficient matrix of the linear system of equations; Let be the objective vector of the system of linear equations; is the parameter vector in the homography matrix.

[0045] Furthermore, S5 specifically includes:

[0046] The registered visible light small receptive field image and infrared small receptive field image are respectively processed by the ResNet50 network to extract features, and the visible light feature map and infrared feature map are output respectively.

[0047] Visible light feature maps and infrared feature maps are input into the CMPA-Net network, which includes a cross-modal cross-attention network and a multi-scale pyramid. The visible light feature maps and infrared feature maps are fused through the cross-modal cross-attention network to obtain a fused feature map. This fused feature map is then used to extract fine-grained and coarse-grained features through the multi-scale pyramid.

[0048] Furthermore, feature fusion is performed using a cross-modal cross-attention network, the process of which is as follows:

[0049] The visible light feature map and the infrared feature map are transformed by a linear feature transformation, which is expressed as follows:

[0050]

[0051]

[0052]

[0053] In the formula, To query features, Key features, Value characteristics; Visible light characteristic map; Infrared feature map; , , They are respectively Q, K, The linear projection matrix;

[0054] Calculate the cross-attention weights and normalize them, as follows:

[0055]

[0056] In the formula, For cross-attention weights, Indicates the activation function; Indicates the scaling factor;

[0057] Feature fusion is based on computationally calculated cross-attention weights, and is expressed as follows:

[0058]

[0059]

[0060] In the formula, These are cross features based on attention weights; Represents the fused feature map; For channel splicing, For convolution kernel; This refers to the offset during the calculation of the fused feature map.

[0061] Furthermore, in S6, the multi-fine-grained identification network includes a species classifier, a multi-fine-grained classification concatenation module, and a growth stage identifier. Its classification and identification process includes the following sub-steps:

[0062] S61. Input the labeled visible light large receptive field image into the species classifier, output the species prediction result, and send the species prediction result to the growth period identifier.

[0063] S62, the multi-fine-grained classification concatenation module transmits the features processed by the category classifier to the growth stage recognizer to achieve information sharing;

[0064] S63. Input the fine-grained features and category prediction results of the fused feature map into the growth period recognizer and output the growth period prediction results.

[0065] Furthermore, in S61, the category classifier includes an improved ResNet network, incorporating an attention mechanism and a Dropout layer into the ResNet network's classification head. Its prediction process is as follows:

[0066] Labeled visible light large receptive field images are input into an improved ResNet network. Global average pooling is used to compress the spatial dimension of the output global features to obtain their global semantics.

[0067]

[0068] In the formula, Represents global semantics; For global average pooling; For global features;

[0069] Global semantics are mapped to the number of categories through two fully connected layers, and then the predicted category probability is output through the Softmax function:

[0070]

[0071]

[0072]

[0073] In the formula, , Global features Intermediate features after ReLU activation and Features after further linear transformation; Indicates the activation function; , These are the weight matrix and bias vector for the linear transformation, respectively; For the first Feature values ​​corresponding to crop types; For the first Feature values ​​corresponding to crop types; Crop category index; To predict the probability of crop category c, the final category prediction is:

[0074]

[0075] In the formula, This is the result of the category prediction.

[0076] Furthermore, S63 specifically includes the following sub-steps:

[0077] S631. A channel attention mechanism is used to perform key feature enhancement processing on the fine-grained features of the fused feature map, which is expressed as follows:

[0078]

[0079]

[0080] In the formula, Represents the channel attention weight map; Indicates the activation function; Represents convolution; To fuse fine-grained features from feature maps; This represents the enhanced fine-grained features; This is channel multiplication;

[0081] The enhanced fine-grained features are then compressed into vectors:

[0082]

[0083] In the formula, This represents the enhanced fine-grained feature vector;

[0084] S632, the multi-fine-grained classification concatenation module integrates global features processed by the category classifier. Mapping to the dimension of shallow feature matching and then upsampling it yields fine-grained features that are integrated with the fused feature map. Global vectors of the same spatial size and the global vector With the enhanced fine-grained feature vector By concatenating channels, a multi-fine-grained fused feature vector is obtained:

[0085]

[0086] In the formula, This represents a multi-fine-grained fused feature vector;

[0087] S633. The One-Hot encoding of the category prediction results is concatenated with multi-fine-grained fusion features to inject category priors into the classifier during the growth period:

[0088]

[0089]

[0090] In the formula, This indicates the encoding method for converting categorical variables into binary vectors. Indicates an indicator function; Conditional fusion features representing prior injection types;

[0091] Will The vector is mapped to the number of growth stages through a fully connected layer. The mapping process is as follows:

[0092]

[0093]

[0094]

[0095] The final growth period prediction is expressed as follows:

[0096]

[0097] In the formula, express Features obtained after linear transformation and ReLU activation; Indicates to The weight matrix for linear mapping; , Both represent offsets; express The features obtained after linear transformation are used for probability calculation of subsequent growth stages; Indicates to The weight matrix for linear mapping; This represents the predicted probability value for the t-th growth stage after Softmax normalization. This represents the characteristic value corresponding to the t-th growth stage; Indicates the first Characteristic values ​​corresponding to each growth stage; Indicates the growth stage index; This indicates the predicted growth period.

[0098] The multi-scale field-view, multi-fine-grained crop identification method provided by this invention has the following beneficial effects:

[0099] This invention can accurately identify crop types and growth stages based on crop images of a region, providing a lightweight foundation for precision agriculture. In subsequent agricultural operations, the drone only needs to take pictures at a fixed altitude to achieve crop classification, eliminating the need for more close-up images, resulting in less data and higher model efficiency.

[0100] The multi-fine-grained recognition network of this invention divides the task into two associated classifiers. The species classifier outputs the predicted species, which serves as a prior condition for the growth period classifier. The results of the species classifier are combined with a channel attention mechanism to achieve the recognition of the growth period. The recognition network that introduces prior conditions also makes the entire model more lightweight, and the recognized crop growth period can provide effective crop information for smart agriculture decision-making. Attached Figure Description

[0101] Figure 1 This is a flowchart of a multi-scale field-of-view crop multi-fine-grained identification method in an embodiment of the present invention.

[0102] Figure 2 This describes the process for constructing a multimodal farmland dataset in an embodiment of the present invention.

[0103] Figure 3 This is a flowchart of the crop multi-scale fine-grained identification method under multi-scale field of view in an embodiment of the present invention. Detailed Implementation

[0104] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0105] This embodiment of the multi-scale field-of-view crop multi-fine-grained identification method can accurately identify crop types and growth stages, providing a lightweight foundation for precision agriculture. (Reference) Figure 1 and Figure 3 Specifically, it includes the following:

[0106] S1. Obtain infrared images and visible light images of farmland crops at different scales, and then construct farmland datasets at different scales;

[0107] For details, please refer to Figure 2The farmland dataset at different scales includes farmland images of varying sizes taken at different camera focal lengths, which can be categorized into large receptive field images and small receptive field images. Large receptive field images refer to data encompassing large crop areas captured with a wide field of view, used for crop species prediction. Small receptive field images refer to images of individual crops or data on crop leaves and stems captured by zooming in. The farmland dataset includes images acquired by visible light cameras and infrared cameras. The acquisition method involves a drone carrying an electro-optical pod flying to a suitable height above the farmland. The visible light and infrared cameras in the pod simultaneously capture farmland data, continuously adjusting the focal length. The final dataset includes multi-scale and multi-modal images.

[0108] In one specific embodiment, a large-scale experimental farmland area was used, with crop types including rice, corn, and wheat. A drone equipped with an electro-optical pod, following a pre-planned flight path, flew over the farmland to collect data. This invention utilizes only the visible light and infrared cameras within the electro-optical pod. The drone flew over the farmland along the pre-planned flight path and collected data from different crop areas, acquiring data at different scales of view. First, large-area crop images were directly captured. Then, the camera's focus could be adjusted at the same location to acquire images with more fine-grained information, including crop leaf texture and more phenotypic features. After data collection, the farmland dataset included both visible light and infrared modalities, containing both global images with a large field of view and detailed images of individual crops with a smaller field of view.

[0109] S2, Preprocessing infrared and visible light images;

[0110] The preprocessing in this embodiment includes data cleaning, radiometric correction, and denoising. Data cleaning is mainly to remove low-quality images; radiometric correction is mainly to eliminate measurement errors caused by factors such as differences in sensor response and changes in lighting conditions; and denoising is mainly to suppress image noise and enhance image quality.

[0111] S3. Label and annotate visible light images at different scales;

[0112] For visible light large receptive field images, which include large areas of crops with little crop detail, they are not suitable for distinguishing growth stages; only species information needs to be labeled. For visible light small receptive field images, which include the morphology of individual crops or information on individual leaves, the growth stage can be identified by the leaf morphology and texture. Therefore, the species of the plant and its growth stage need to be labeled according to scientific agricultural knowledge.

[0113] S4. Register visible images at different scales, and then register the registered visible light images with infrared images;

[0114] The data in this embodiment requires a two-step registration operation. First, the visible light small receptive field image and the next visible light large receptive field image are registered. The two images need to be taken at different focal lengths in the same scene, so that their spatial coordinates are aligned. This determines that the crop in the visible light small receptive field image is aligned with a certain coordinate in the visible light large receptive field image, that is, it is determined that it is a crop belonging to this area. This allows the identification of the crop species information in the small receptive field image, which then provides prior conditions for multi-fine-grained identification and prediction of the growth period of the corresponding species. Second, after the coordinate alignment, the visible light small receptive field image is registered with the infrared small receptive field image to prepare for multimodal fusion.

[0115] Both registration operations in this embodiment are based on the homography registration method of ORB features. By comparing the spatial positional relationship of feature points, the transformation parameters such as translation, rotation and scaling between images are calculated to achieve accurate alignment in the spatial dimension and ensure that the spatial position of ground features reflected by different types of images is consistent.

[0116] Specifically, the homography registration method based on ORB features is used to register the visible light large receptive field image and the visible light small receptive field image. Then, the same homography registration method based on ORB features is used to register the registered visible light small receptive field image with the infrared small receptive field image.

[0117] Since the two-step registration method in this embodiment is exactly the same, only the registration of a visible light large receptive field image and a visible light small receptive field image will be used as an example for explanation. The method includes two steps: ORB feature detection and matching, and homography matrix estimation. Using the matched feature points, the transformation relationship between the two images, i.e., the homography matrix, can be calculated. Using the homography matrix, the four corner points of the query image can be mapped onto the scene image, thereby locating the target. Specifically, this includes the following sub-steps:

[0118] S41, ORB Feature Detection and Matching;

[0119] ORB uses the FAST (features from accelerated segment test) algorithm to detect feature points. This algorithm is based on the grayscale values ​​of the image surrounding a feature point. It detects the pixel values ​​around a candidate feature point; if a candidate point has a sufficiently large number of pixels in its neighborhood with significantly different grayscale values, it is considered a feature point. The detected ORB feature points are then selected from the FAST feature points using the Harris corner metric, choosing the N feature points with the largest Harris corner response values. After obtaining the feature points, their attributes need to be described. The output of these attributes is called the feature descriptor (Feature DescritorS). ORB uses the BRIEF algorithm to calculate the descriptor of a feature point. The core idea of ​​the BRIEF algorithm is to select N pairs of points around a keypoint P in a certain pattern, and combine the comparison results of these N pairs as the descriptor.

[0120] During ORB feature matching, for each feature point in a small receptive field image, the two closest feature points are found in the large receptive field image. The similarity of ORB feature point descriptors is measured using Hamming distance; a smaller value indicates greater similarity. Let the feature point matching condition be:

[0121]

[0122] In the formula, The Hamming distance is the nearest neighbor. The Hamming distance to the next nearest neighbor; This threshold value is a key parameter used to filter valid matches and eliminate noise. The effect of the threshold varies at different stages, directly impacting registration accuracy and robustness. If the ratio of the nearest distance to the second nearest distance is less than this threshold, the match is retained and called an inlier; otherwise, it is considered an incorrect match and called an outlier. Here, the threshold is set to [value missing]. =0.7, strictly screening matching pairs to ensure matching quality;

[0123] S42. Homography matrix estimation;

[0124] Homography matrix (denoted as ) H ) is a 3×3 matrix that describes the "projection transformation relationship between two planes". The core of its transformation formula is to map the pixels of the small receptive field image to the corresponding positions of the large receptive field image through homogeneous coordinates.

[0125] Based on the homography matrix, pixels in a visible light small receptive field image are mapped to their corresponding positions in a visible light large receptive field image. The coordinate transformation is expressed as follows:

[0126]

[0127] In the formula, These are the homogeneous coordinates of a visible light small receptive field image; The homogeneous coordinates of the visible light large receptive field image; It is a homography matrix. These are normalization parameters;

[0128] homography matrix H The standard form is:

[0129]

[0130] In step S42, the RANSAC algorithm is used to select interior points from feature point pairs containing multiple exterior points. Then, based on the selected interior points, the least squares method is used to solve for the optimal homography matrix. This specifically includes the following steps:

[0131] S421, Initial Matching Pairs:

[0132]

[0133] In the formula, The homogeneous coordinates of the first feature point involved in matching in the visible light small receptive field image are denoted as . ; This represents the homogeneous coordinates of the first feature point in the visible light large receptive field image that participates in matching; This represents the homogeneous coordinates of the nth feature point participating in the matching process in the visible light small receptive field image and the visible light large receptive field image;

[0134] Four pairs of non-collinear points are randomly selected from the initial matched point pairs. , , , , These represent the first, second, third, and fourth randomly selected feature point pairs, used to solve for the temporary homography matrix. ;

[0135] S422, Based on 4 pairs of non-collinear points A system of linear equations with four pairs of points (two equations per pair, for a total of eight equations) is constructed using the least squares method, and the temporary homography matrix is ​​solved.

[0136]

[0137] In the formula, This is a temporary homography matrix; The parameter represents the parameter that makes the function reach its minimum value; Indicates the first randomly selected... i For feature point pairs ; , Indicates the first i Homogeneous coordinates of the point; Euclidean distance;

[0138] S423, Interior Point Statistics: Calculate the reprojection error of matched point pairs based on the temporary homography matrix.

[0139]

[0140] In the formula, This is due to reprojection error; , represents the coordinates of feature points in the large receptive field image and the small receptive field image; w is the third dimension of the homogeneous coordinates of the feature points;

[0141] If reprojection error <Error threshold Then, the corresponding points are marked as interior points, and the set of interior points is denoted as . ;

[0142] S424. Repeat S421~S423 several times, retain the set of interior points with the largest number of interior points, and denote it as the optimal set of interior points. And at the same time retain the corresponding temporary homography matrix;

[0143] S425. For the optimal set of interior points The homogeneous coordinate transformation of the interior points satisfies:

[0144]

[0145] After expanding the homogeneous coordinates of the interior point, it is transformed into a system of linear equations:

[0146]

[0147] Solve this system of linear equations using the least squares method:

[0148]

[0149] In the formula, Scale factor; This is the coefficient matrix of the linear system of equations; Let be the objective vector of the system of linear equations; Given the parameter vectors in the homography matrix, solving for h yields the homography matrix. H .

[0150] S5. The registered visible light image and infrared image are fused to obtain a fused feature map;

[0151] This embodiment employs the CMPA-Net network for multimodal image fusion. The CMPA-Net network primarily includes a cross-modal cross-attention network and a multi-scale feature pyramid. The cross-modal cross-attention network actively mines the relationships between different modalities, thereby generating more discriminative fusion features. The multi-scale feature pyramid extracts features from different network depths, constructing a top-down low-resolution semantic layer through downsampling and a bottom-up high-resolution detail layer through upsampling and fusion, ultimately forming a feature hierarchy covering multiple scales. This results in more detailed features, which is more beneficial for recognition during the growth stage.

[0152] Specifically, visible light small receptive field images can provide intuitive details of crop appearance, such as color, texture, and lesion shape, while infrared small receptive field images can provide hidden physiological conditions, such as temperature and moisture, ultimately achieving comprehensive crop status monitoring from the surface to the interior. The fusion part uses a cross-modal cross-attention mechanism and an adaptive weight fusion method. Images after registration have spatial consistency, but the feature value ranges of the two modalities may differ greatly. Normalization is used to normalize the feature values ​​to a similar range, and then feature extraction is performed on the single-modal data. Two independent feature extraction methods are used to process the visible light and infrared data respectively, obtaining their respective high-dimensional features.

[0153] Specifically, features are extracted from the registered visible light small receptive field image and infrared small receptive field image using a ResNet50 network, and visible light feature maps are output separately. and infrared feature map ;in, , For the number of channels, The size of the feature map.

[0154] Subsequently, the visible light feature map and the infrared feature map are input into the CMPA-Net network, which includes a cross-modal cross-attention network and a multi-scale pyramid. The visible light feature map and the infrared feature map are fused through the cross-modal cross-attention network to obtain a fused feature map. This fused feature map is then used to extract fine-grained and coarse-grained features through the multi-scale pyramid.

[0155] The process of feature fusion using a cross-modal cross-attention network is as follows:

[0156] The visible light feature map and the infrared feature map are subjected to feature linear transformation, which includes three parts: feature linear transformation, cross attention weight calculation and fusion feature generation.

[0157] Among them, the characteristic linear transformation uses visible light as the query. Q Infrared as key K / valueV Specifically, it is expressed as:

[0158]

[0159]

[0160]

[0161] In the formula, To query features, Key features, Value characteristics; Visible light characteristic map; Infrared feature map; , , They are respectively Q, K , The linear projection matrix; , , They are respectively Q, K, V The number of channels.

[0162] Calculate the cross-attention weights and normalize them, as follows:

[0163]

[0164] In the formula, For cross-attention weights, Indicates the activation function; Indicates the scaling factor;

[0165] Feature fusion is based on computationally calculated cross-attention weights, and is expressed as follows:

[0166]

[0167]

[0168] In the formula, These are cross features based on attention weights; Represents the fused feature map; For channel splicing, For convolution kernel; This refers to the offset during the calculation of the fused feature map.

[0169] Multi-scale pyramids, from fused feature maps Fine-grained and coarse-grained features are extracted. The multi-scale feature pyramid is mainly divided into downsampling and upsampling. Basic scale features are extracted from top to bottom. Upsampling performs bilinear interpolation on the top-level features by 2 times and connects them horizontally with the feature layer of the same resolution, thus fusing high-level semantics and low-level detailed features.

[0170] S6. Input the fused feature map and the labeled visible light image into a multi-fine-grained recognition network for classification and recognition, and output the crop type and growth stage;

[0171] This embodiment designs a multi-fine-grained recognition network (MGR-Net) to achieve multi-fine-grained recognition of crop species and growth stages. The entire network includes a species classifier and a growth stage identifier, which are connected by a multi-fine-grained classification concatenation module. First, the fused small receptive field image and the visible light large receptive field image with label information are input into the entire multi-fine-grained recognition network (MGR-Net) for training, and then tested. The visible light large receptive field image serves as the input to the species classifier, and the output result is the crop species in this field of view; the output result of the species classifier provides prior conditions for the growth stage classifier, and together with the features corresponding to the fused small receptive field image, it serves as the input to the growth stage classifier, and the output result is the growth stage of the crop.

[0172] Its classification and recognition process includes the following steps:

[0173] S61, Category Classifier;

[0174] The crop classification network aims to classify crops from images with a large receptive field. Its input is a visible light image with a large receptive field, and its output is the crop type. The classifier uses a modified ResNet network to perform the classification task. The improvement mainly lies in the custom classification head. An attention mechanism is used in the classification head to help the model focus on the crop itself, highlighting the crop area and reducing background and soil interference. A Dropout layer is then added to prevent overfitting. The output features are compressed in dimensionality through a fully connected layer, and then softmax is used to output the class probabilities. The classifier takes global features. Global average pooling (GAP) is used to compress the spatial dimension while preserving global semantics.

[0175]

[0176] In the formula, Represents global semantics; For global average pooling; For global features;

[0177] Global semantics are mapped to the number of categories through two fully connected layers, and then the predicted category probability is output through the Softmax function:

[0178]

[0179]

[0180]

[0181] In the formula, , Global features Intermediate features after ReLU activation and Features after further linear transformation; Indicates the activation function; , These are the weight matrix and bias vector for the linear transformation, respectively; For the first Feature values ​​corresponding to crop types; For the first Feature values ​​corresponding to crop types; Crop category index; To predict the probability of crop category c, the final category prediction is:

[0182]

[0183] In the formula, This is the result of the category prediction.

[0184] S62, Multi-fine-grained classification concatenation module; The multi-fine-grained classification concatenation module transmits the features processed by the category classifier to the growth stage recognizer to achieve information sharing;

[0185] Specifically, when designing a multi-fine-grained classification concatenation module between the crop classifier and the growth stage identifier, the core objective is to achieve effective feature transfer and sharing while preserving the specificity of the two tasks. Crop crop classification and growth stage identification are strongly correlated, allowing them to jointly participate in growth stage identification by directly concatenating crop features and basic visual features. The multi-fine-grained classification concatenation module inputs the global feature P1 extracted by the crop classifier into the growth stage identifier and preprocesses it. A convolution module unifies the number of channels, and P1 is upsampled to make its spatial size the same as the shallow features input to the growth stage identifier. The global feature is then used as an external input to the growth stage identifier and fused with shallow detail features through feature mapping.

[0186] S63, Growth Stage Recognizer: Input the fine-grained features and category prediction results of the fused feature map into the growth stage recognizer, and output the growth stage prediction results;

[0187] The growth period identifier in this embodiment aims to perform further fine-grained classification of crops after the crop type has been determined. The growth period identifier mainly relies on the fine-grained features of the crop for prediction. Since the growth period of each crop is different, the prediction of the growth period identifier needs to be combined with the type prediction results to narrow down the judgment range. The prediction task is mainly divided into three steps: fine-grained feature extraction, type prediction result fusion, and growth period prediction. Its identification process specifically includes the following steps:

[0188] S631. A channel attention mechanism is used to perform key feature enhancement processing on the fine-grained features of the fused feature map, which is expressed as follows:

[0189]

[0190]

[0191] In the formula, Represents the channel attention weight map; Indicates the activation function; Represents convolution; To fuse fine-grained features from feature maps; This represents the enhanced fine-grained features; This is channel multiplication;

[0192] The enhanced fine-grained features are then compressed into vectors:

[0193]

[0194] In the formula, This represents the enhanced fine-grained feature vector;

[0195] S632, the multi-fine-grained classification concatenation module integrates global features processed by the category classifier. Mapping to the dimension of shallow feature matching and then upsampling it yields fine-grained features that are integrated with the fused feature map. Global vectors of the same spatial size and the global vector With the enhanced fine-grained feature vector By concatenating channels, a multi-fine-grained fused feature vector is obtained:

[0196]

[0197] In the formula, This represents a multi-fine-grained fused feature vector;

[0198] S633. The One-Hot encoding of the category prediction results is concatenated with multi-fine-grained fusion features to inject category priors into the classifier during the growth period:

[0199]

[0200]

[0201] In the formula, This indicates the encoding method for converting categorical variables into binary vectors; Indicates an indicator function; Conditional fusion features representing prior injection types;

[0202] Will The vector is mapped to the number of growth stages through a fully connected layer. The mapping process is as follows:

[0203]

[0204]

[0205]

[0206] The final growth period prediction is expressed as follows:

[0207]

[0208] In the formula, express Features obtained after linear transformation and ReLU activation; Indicates to The weight matrix for linear mapping; , Indicates the offset; express The features obtained after linear transformation are used for probability calculation of subsequent growth stages; Indicates to The weight matrix for linear mapping; This represents the predicted probability value for the t-th growth stage after Softmax normalization. This represents the characteristic value corresponding to the t-th growth stage; Indicates the first Characteristic values ​​corresponding to each growth stage; Indicates the growth stage index; This indicates the predicted growth period.

[0209] Although specific embodiments of the invention have been described in detail with reference to the accompanying drawings, this should not be construed as limiting the scope of protection of this patent. Various modifications and variations that can be made by a person skilled in the art without inventive effort within the scope described in the claims still fall within the scope of protection of this patent.

Claims

1. A method for identifying crops at multiple fine-grained scales under a multi-scale field of view, characterized in that, Includes the following steps: S1. Acquire infrared images and visible light images of farmland crops at different scales; S2. Preprocess infrared and visible light images; S3. Label and annotate visible light images at different scales; S4. Register visible images at different scales, and then register the registered visible light images with infrared images; S5. The registered visible light image and infrared image are fused to obtain a fused feature map; S6. Input the fused feature map and the labeled visible light image into the multi-fine-grained recognition network for training. Then, input infrared images and visible light images of farmland crops at different scales into the trained multi-fine-grained recognition network for prediction. The output is the crop type and growth stage. In S6, the multi-fine-grained identification network includes a species classifier, a multi-fine-grained classification concatenation module, and a growth stage identifier. Its classification and identification process includes the following sub-steps: S61. Input the labeled visible light large receptive field image into the species classifier for training, then input the visible light large receptive field image to be predicted into the trained species classifier, output the species prediction result, and send the species prediction result to the growth period identifier. S62, the multi-fine-grained classification concatenation module transmits the features processed by the category classifier to the growth stage recognizer to achieve information sharing; S63. Input the fine-grained features and category prediction results of the fused feature map into the growth period recognizer for training. Then, input the visible light small receptive field image and the infrared small receptive field image to be predicted into the trained growth period recognizer and output the growth period prediction result.

2. The method for identifying crops at multiple fine-grained scales under a multi-scale field of view according to claim 1, characterized in that, In S1, infrared images of different scales include infrared large receptive field images and infrared small receptive field images; visible light images of different scales include visible light large receptive field images and visible light small receptive field images.

3. The method for identifying crops at multiple fine-grained scales under a multi-scale field of view according to claim 2, characterized in that, In step S3, the visible light images at different scales are labeled, including: Manual labeling was used to label species in visible light images with large receptive fields, and to label species and growth stage in visible light images with small receptive fields.

4. The method for identifying crops at multiple fine-grained scales under a multi-scale field of view according to claim 2, characterized in that, In step S4, the homography registration method of ORB features is used to register the visible light large receptive field image and the visible light small receptive field image, and then the same ORB feature homography registration method is used to register the registered visible light small receptive field image with the infrared small receptive field image. The homography registration method using ORB features is used to register visible light large receptive field images and visible light small receptive field images. This process includes the following steps: S41. Hamming distance is used to measure the similarity of ORB feature point descriptors in order to match feature points; The matching conditions for feature points are as follows: In the formula, The Hamming distance is the nearest neighbor. The Hamming distance to the next nearest neighbor; Indicates the threshold; S42. Based on the homography matrix, the pixels of the visible light small receptive field image are mapped to the corresponding positions in the visible light large receptive field image. The coordinate transformation is expressed as follows: In the formula, These are the homogeneous coordinates of a visible light small receptive field image; The homogeneous coordinates of the visible light large receptive field image; It is a homography matrix.

5. The method for identifying crops at multiple fine-grained scales under a multi-scale field of view according to claim 4, characterized in that, In step S42, the RANSAC algorithm is used to select interior points from feature point pairs containing multiple exterior points. Then, based on the selected interior points, the least squares method is used to solve for the optimal homography matrix. This specifically includes the following steps: S421. Randomly select 4 pairs of non-collinear points from the initial matched point pairs. , , , , These represent the first, second, third, and fourth randomly selected feature point pairs, respectively. S422, Based on 4 pairs of non-collinear points A system of linear equations for four pairs of points is constructed using the least squares method, and the temporary homography matrix is ​​solved: In the formula, This is a temporary homography matrix; The parameter represents the parameter that makes the function reach its minimum value; Indicates the first randomly selected... i For feature points; , Indicates the first i Homogeneous coordinates of the point; Euclidean distance; S423. Calculate the reprojection error of the matching point pair based on the temporary homography matrix: In the formula, This is for reprojection error; , represents the coordinates of feature points in the large receptive field image and the small receptive field image of visible light, respectively; w is the third dimension of the homogeneous coordinates of the feature points; If reprojection error <Error threshold Then, the corresponding points are marked as interior points, and the set of interior points is denoted as . ; S424. Repeat S421~S423 several times, retain the set of interior points with the largest number of interior points, and denote it as the optimal set of interior points. And at the same time retain the corresponding temporary homography matrix; S425. For the optimal set of interior points The homogeneous coordinate transformation of the interior points satisfies: After expanding the homogeneous coordinates of the interior point, it is transformed into a system of linear equations: Solve this system of linear equations using the least squares method: In the formula, Scale factor; This is the coefficient matrix of the linear system of equations; Let be the objective vector of the system of linear equations; is the parameter vector in the homography matrix.

6. The method for identifying crops at multiple fine-grained scales under a multi-scale field of view according to claim 2, characterized in that, S5 specifically includes: The registered visible light small receptive field image and infrared small receptive field image are respectively processed by the ResNet50 network to extract features, and the visible light feature map and infrared feature map are output respectively. Visible light feature maps and infrared feature maps are input into the CMPA-Net network, which includes a cross-modal cross-attention network and a multi-scale pyramid. The visible light feature maps and infrared feature maps are fused through the cross-modal cross-attention network to obtain a fused feature map. This fused feature map is then used to extract fine-grained and coarse-grained features through the multi-scale pyramid.

7. The method for identifying crops at multiple fine-grained scales under a multi-scale field of view according to claim 6, characterized in that, The process of feature fusion using a cross-modal cross-attention network is as follows: The visible light feature map and the infrared feature map are transformed by a linear feature transformation, which is expressed as follows: In the formula, To query features, Key features, Value characteristics; Visible light characteristic map; Infrared feature map; , , They are respectively Q, K, The linear projection matrix; Calculate the cross-attention weights and normalize them, as follows: In the formula, For cross-attention weights, Indicates the activation function; Indicates the scaling factor; Feature fusion is based on computationally calculated cross-attention weights, and is expressed as follows: In the formula, These are cross features based on attention weights; Represents the fused feature map; For channel splicing, For convolution kernel; This refers to the offset during the calculation of the fused feature map.

8. The method for identifying crops at multiple fine-grained scales under a multi-scale field of view according to claim 1, characterized in that, In step S61, the category classifier includes an improved ResNet network, incorporating an attention mechanism and a Dropout layer into the ResNet network's classification head. Its prediction process is as follows: Labeled visible light large receptive field images are input into an improved ResNet network. Global average pooling is used to compress the spatial dimension of the output global features to obtain their global semantics. In the formula, Represents global semantics; For global average pooling; For global features; Global semantics are mapped to the number of categories through two fully connected layers, and then the predicted category probability is output through the Softmax function: In the formula, , Global features Intermediate features after ReLU activation and Features after further linear transformation; Indicates the activation function; , These are the weight matrix and bias vector for the linear transformation, respectively; For the first Feature values ​​corresponding to crop types; For the first Feature values ​​corresponding to crop types; Crop category index; To predict the probability of crop category c, the final category prediction is: In the formula, This is the result of the category prediction.

9. The method for identifying crops at multiple fine-grained scales under a multi-scale field of view according to claim 8, characterized in that, S63 specifically includes the following steps: S631. A channel attention mechanism is used to perform key feature enhancement processing on the fine-grained features of the fused feature map, which is expressed as follows: In the formula, Represents the channel attention weight map; Indicates the activation function; Represents convolution; To fuse fine-grained features from feature maps; This represents the enhanced fine-grained features; This is channel multiplication; The enhanced fine-grained features are then compressed into vectors: In the formula, This represents the enhanced fine-grained feature vector; S632, the multi-fine-grained classification concatenation module integrates global features processed by the category classifier. Mapping to the dimension of shallow feature matching and then upsampling it yields fine-grained features that are integrated with the fused feature map. Global vectors with the same spatial size and the global vector With the enhanced fine-grained feature vector By concatenating channels, a multi-fine-grained fused feature vector is obtained: In the formula, This represents a multi-fine-grained fused feature vector; S633. The One-Hot encoding of the category prediction results is concatenated with multi-fine-grained fusion features to inject category priors into the classifier during the growth period: In the formula, This indicates the encoding method for converting categorical variables into binary vectors. Indicates an indicator function; Conditional fusion features representing prior injection types; Will The vector is mapped to the number of growth stages through a fully connected layer. The mapping process is as follows: The final growth period prediction is expressed as follows: In the formula, express Features obtained after linear transformation and ReLU activation; Indicates to The weight matrix for linear mapping; , Both represent offsets; express The features obtained after linear transformation are used for probability calculation of subsequent growth stages; Indicates to The weight matrix for linear mapping; This represents the predicted probability value for the t-th growth stage after Softmax normalization. This represents the characteristic value corresponding to the t-th growth stage; Indicates the first Characteristic values ​​corresponding to each growth stage; Indicates the growth stage index; This indicates the predicted growth period.

Images