Image semantic segmentation method and system for wheat seedlings in the tillering stage

Abstract

The present invention discloses a method and system for image semantic segmentation of wheat seedlings in the tillering stage, the method comprising the steps of: S1, collecting multiple first images; S2, pre-processing based on color correction, super-green segmentation and anisotropic filtering algorithms, annotating and fine-tuning each mask image; S3, constructing a segmentation network model with Segformer as the backbone and training it; S4, calculating each evaluation index; S5, correcting each spectral image and obtaining each phenotypic parameter through feature matching and mask resampling. The system is used to support the implementation of the above method. The present invention improves the image quality and ensures the accuracy of subsequent segmentation by improving the pre-processing based on color correction, super-green segmentation and improved anisotropic diffusion filtering; also uses an improved segmentation network model and adds multiple modules to improve the feature extraction capability; and eliminates background interference and calculates multiple phenotypic parameters through multiple transformations and feature matching, making the evaluation results more comprehensive.

Description

Technical Field

[0001] The present invention relates to the field of agricultural technology, and in particular to an image semantic segmentation method and system for wheat seedlings in the tillering stage. Background Art

[0002] Traditional methods for calculating wheat seedling area suffer from low accuracy, poor efficiency, and insufficient generalization in complex field environments. Typically, wheat seedling images contain background interference such as soil, and the seedlings themselves possess numerous detailed features. Therefore, using wheat seedling images to accurately segment the seedlings from the background and generate quantitative indicators of seedling biomass can lay a solid foundation for crop growth assessment and genetic improvement.

[0003] In existing wheat image segmentation, semantic segmentation networks are often used to segment and identify wheat plots or ears during the maturing stage. However, there is no mature semantic segmentation network suitable for processing wheat seedlings in the tillering stage. Furthermore, in practical applications, farmland environments are complex. Because wheat seedlings are small, interference from background soil and weeds can significantly interfere with them. If existing semantic segmentation networks for mature wheat are rigidly applied for segmentation and recognition, the accuracy rate will be very low. Furthermore, there is a lack of application systems suitable for identifying wheat seedlings during the tillering stage.

[0004] That is to say, the measures used in the prior art include at least the following three problems:

[0005] 1) Traditional semantic segmentation networks are only suitable for mature wheat and cannot be directly applied to the analysis of wheat seedlings during the tillering stage;

[0006] 2) During the tillering stage, wheat seedlings are small and are greatly disturbed by background soil and weeds, resulting in very low recognition accuracy.

[0007] 3) The lack of a supporting system makes actual testing very inconvenient. Summary of the Invention

[0008] In response to the above three problems, the purpose of the present invention is to propose an image semantic segmentation method and system for wheat seedlings in the tillering period. First, through color correction based on linear regression, super green segmentation and improved anisotropic diffusion filtering, the quality of the input image is significantly improved during preprocessing to ensure the accuracy and reliability of subsequent segmentation tasks; an improved segmentation network model is also used, and a variety of modules are added for improvement to further enhance the feature extraction capability of the model; and through a variety of transformations and feature matching, background interference is effectively eliminated and multiple parameters are calculated for display, making the evaluation and prediction more comprehensive; in addition, a system is provided to support the operation of this method, which is convenient for actual operation and greatly enhances convenience.

[0009] This is achieved through the following technical solutions:

[0010] First, a semantic segmentation method for images of wheat seedlings at the tillering stage is proposed, which includes the following steps:

[0011] S1. Setting up multiple calibration frames in multiple areas of a wheat field, with a color calibration plate set up in every 10 areas, and using a drone to collect multiple first images of the wheat seedlings, the first images including RGB images and hyperspectral images;

[0012] S2. Preprocess each RGB image in step S1 using a linear regression color correction algorithm to obtain each color-corrected second image; then calculate each second image using a super-green segmentation algorithm, retain each pixel that meets the segmentation threshold and set each pixel that does not meet the segmentation threshold to 0, to obtain each third image; continue to use an isotropic filtering algorithm to patch the edges of each third image to obtain each mask image; then use annotation software to annotate and adjust the details of each mask image;

[0013] S3. Perform semantic segmentation on each mask image after detail adjustment in step S2 and each first image in step S1, including: scale division, parameter adjustment, construction of a segmentation network model based on Segformer as the backbone, model improvement, and model training; wherein the model improvement includes: optimizing the segmentation network model based on a spatial and channel collaborative attention module, a cross-layer feature pyramid module, and a dilated spatial convolution pooling pyramid module;

[0014] S4. After obtaining the results of the model training in step S3, for each mask image after detail adjustment, calculate the corresponding evaluation index; the evaluation index includes row seedling area and density;

[0015] S5. First, estimate each transformation matrix from each spectral image to the corresponding RGB image through feature matching, and control the alignment of each spectral image with the corresponding RGB image based on each transformation matrix; continue to perform mask resampling to correct each spectral image, and then calculate the various spectral indices of the wheat seedlings, and display the various spectral indices and each evaluation indicator in step S4 as phenotypic parameters.

[0016] Preferably, in step S1, a 0.33 m × 0.33 m calibration frame is set up for every 5 areas; each hyperspectral image includes four spectral bands of NIR, R, G and RE; the NIR spectral band represents the near-infrared band, the R spectral band is the infrared band, the G spectral band is the green band, and the RE spectral band is the red-edge band; before using a drone to collect multiple first images, select the afternoon time period with cloud cover ≤8%, relative humidity ≤35% and visibility >15 meters for collection, and the afternoon time period is 12:00-14:00; when using a drone for collection, set the drone's flight altitude to 15 meters and the camera's lens angle to the ground to 90 degrees.

[0017] Preferably, the steps of color correction in step S2 are as follows: I) calculating the maximum and minimum values ​​of the row and column pixel positions in the sub-area where each color correction plate is located, and setting the number of interpolation points to 6 and 4 in the column direction and row direction respectively, and performing linear interpolation to divide the color correction plate in each RGB image into 24 color blocks, and taking the point with the largest pixel value in each color block as the pixel value representative of the corresponding color block; II) performing a [1, R, G, B, RG, RB, BG, RR, BB, GG] polynomial combination on the RGB values ​​of the 24 color blocks in each RGB image to obtain each matrix Vf; then the RGB values ​​of the 24 color blocks of the standard colorimetric card are combined into a matrix C of [3,24], and an objective function is constructed to represent the difference between each matrix Vf and the matrix C. When the sum of the squares of the objective function is controlled to be minimum, the optimal transformation matrix A is obtained; where 1 is a constant term, R, G, and B correspond to the original values ​​of the red, green, and blue color channels respectively, and RG, RB, BG, RR, BB, and GG correspond to the product of two color channels respectively; III) Each corresponding RGB image is converted using each transformation matrix A to obtain each second image after color correction.

[0018] Preferably, when calculating each second image using the super green segmentation algorithm in step S2, the ExG value of each second image is calculated and the segmentation threshold is set to 25, each pixel point with ExG>25 is regarded as the pixel point where the wheat seedlings are located and retained, and each pixel point with ExG≤25 is regarded as a pixel point that does not belong to the wheat seedlings and is set to 0.

[0019] Preferably, in step S2, the step of using an isotropic filtering algorithm to repair the edge of any third image is as follows:

[0020] 1) Traverse each non-zero pixel in the third image, use the convolution kernel of the Sobel operator for each non-zero pixel at position (i, j), and calculate the gradient amplitude of the brightness change rate in the four directions of horizontal left (i-1, j), horizontal right (i+1, j), vertical up (i, j-1), and vertical down (i, j+1) , subscripts W, E, N, and S correspond to left, right, top, and bottom respectively; where i and j are used to represent coordinates;

[0021] 2) The Perona-Malik model is used to control the anisotropy in the horizontal and vertical directions in step 1), and the diffusion coefficient λ in each of the four directions is determined. The diffusion coefficient equation in each direction is constructed and used to calculate the diffusion coefficient of each pixel. The gradients in the four directions of N, S, E, and W , and then use the edge stop function The gradient in each direction is weighted, and the weighted result in each direction is multiplied by the corresponding diffusion coefficient λ and then accumulated to obtain the update amount of each corresponding pixel point, which is used to calculate the update amount of each pixel point after diffusion. The value of and iterate, the stopping condition of the iteration is , represents the gradient energy threshold and is set to 0.1.

[0022] Preferably, in step S3, the ratio division includes: taking each first image and each mask image after detail adjustment as model input, and dividing them into a training set, a test set, and a validation set in a ratio of 7:2:1; the parameter adjustment includes: using the AdamW optimizer to adjust the parameters; when adjusting the parameters, setting the learning rate to 6×10 -5 , the learning decay coefficient is 0.8, the number of samples per batch iteration is 8, and the number of training rounds is 200.

[0023] Preferably, when improving the model in step S3, a spatial and channel collaborative attention module is added to the encoder head of the segmentation network model, a cross-layer feature pyramid module is added after the spatial and channel collaborative attention module, and then a void spatial convolution pooling pyramid is added after the multi-layer perceptron of the decoder in the segmentation network model; wherein the spatial and channel collaborative attention module includes: a spatial attention branch submodule and a channel attention branch submodule; in the spatial attention branch submodule, global average pooling and global maximum pooling are first performed on the input feature map along the channel dimension to generate two spatial descriptors; then the two descriptors are merged by a splicing operation, and processed by a convolution layer and an activation function, and finally a normalized spatial attention weight is output; in the channel attention branch submodule, a channel descriptor is generated by global average pooling and a channel attention weight is generated by a fully connected layer, the weight is multiplied channel by channel with the input feature in the input feature map in vector form, and the normalized channel attention weight is output; the average of the normalized spatial attention weight and the normalized channel attention weight is calculated as the output;

[0024] In the cross-layer feature pyramid module, the input is first divided into 4 groups of different multi-level feature maps, which are horizontally connected through the cross-layer feature integration sub-module to upsample the high-level features of each group of multi-level feature maps to a low-level resolution; then, it enters the cross-layer attention sub-module, which is equipped with a cross-channel attention mechanism and a spatial channel attention mechanism; in the cross-channel attention mechanism, any input feature is decomposed into three feature matrices including the query vector, key vector and value vector through a linear transformation, and then the dot product of the transposed matrix of the query vector and the key vector is calculated, and normalized by the scaling factor, and the softmax activation function is applied to the normalized result to generate the attention weight And the weighted summation with the value vector is performed to obtain the features after channel attention enhancement; in the spatial channel attention mechanism, the features after channel attention enhancement are spliced ​​with any of the aforementioned input features to generate spatial-channel mixed attention features, which are used to arrange the output results from high to low resolution; finally, through lateral connection and upsampling operations, a new pyramid structure is constructed, and the spatial-channel mixed attention features of the new pyramid structure are input into the shared detection head of the segmentation network model to obtain the tensor for model training; the void spatial convolution pooling pyramid includes 4 void convolutions, which are used for convolution dimensionality reduction and output multi-scale feature maps to the ClassSeg convolution of the segmentation network model. The ClassSeg convolution is used to generate segmentation masks during model training.

[0025] Preferably, when calculating each corresponding evaluation index for any mask image that has been adjusted for details in step S4, first extract the red part corresponding to the calibration frame from the mask image and extract the edge contour of the correction frame; then use perspective transformation to map the pixel points in the four corners of the correction frame to a new image of the same size as the mask image, divide the number of mapped pixel points by the actual area of ​​the correction frame to obtain a proportional factor, and then multiply the number of mask pixels in each row of wheat seedlings by the proportional factor to obtain the row seedling area of ​​each row of wheat seedlings; then, use depth-first search-based clustering for the mask area of ​​the wheat seedlings in the mask image, calculate the minimum convex polygon of the enclosing contour of each cluster cluster, calculate the average value of the ratio of the pixel points contained in all cluster clusters and the pixel points contained in the minimum contour polygon, and use the average value as the density.

[0026] Preferably, in step S5, when performing feature matching, the four corners of the correction frame in each RGB image after detail adjustment and each corresponding spectral image are first used as high-confidence matching point pairs, and the RANSAC algorithm is used to estimate the transformation matrix of each spectral image to the corresponding RGB image, and the images are aligned by perspective transformation; when performing mask resampling, each mask image is resampled according to the resolution ratio of each spectral image and each corresponding RGB image, and after resampling to the same mask size in each corresponding spectral image, each resampled mask image is applied to each corresponding spectral image, and the average value of the soil partial reflectance of each pixel point in the soil area in the four spectral bands of NIR, R, G, and RE is calculated, and the original reflectance of each pixel point at the four spectral wavelengths is subtracted from the average value to obtain the reflectance of the wheat seedlings after eliminating the soil reflection in each spectral band. At the same time, the four spectral indices of the wheat seedlings are calculated: normalized difference vegetation index, normalized difference red edge vegetation index, chlorophyll index, and vegetation coverage.

[0027] In addition, a system is proposed, the application interface of which includes a functional part and a display part. The functional part includes a data loading module, a category adding module, an image annotation and detail fine-tuning module, a parameter calculation module, a spectral image annotation module and a data export module, and the display part is used to display the results of each module; wherein, the data loading module is used to obtain each image by selecting a specified image folder; the category adding module is used to input the specified category name and set the annotation and mask color; the image annotation and detail fine-tuning module is used to record and execute the transformation operation of each matrix, annotate and adjust the details of each mask image; the spectral image annotation module is used to perform spectral image annotation, feature matching and overlay each resampled mask image onto each corresponding spectral image; the parameter calculation module is used to support the calculation of row seedling area, density, normalized difference vegetation index, normalized difference red edge vegetation index, chlorophyll index and vegetation coverage; the data export module is used to generate each mask image and each corresponding .json file and the folder location where the results are saved.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] The technical solution of the present invention first significantly improves the quality of the input image during preprocessing through color correction based on linear regression, super-green segmentation and improved anisotropic diffusion filtering, ensuring the accuracy and reliability of subsequent segmentation tasks; it also uses an improved segmentation network model and adds multiple modules for improvement to further enhance the feature extraction capability of the model; it also uses multiple transformations and feature matching to effectively eliminate background interference and calculate multiple parameters for display, making the evaluation and prediction more comprehensive; in addition, a system is provided to support the operation of this method, which facilitates practical operation and greatly enhances convenience. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a flowchart of a method for image semantic segmentation of wheat seedlings in the tillering stage;

[0031] Figure 2 Schematic diagram of collecting images for drones;

[0032] Figure 3 This is a diagram showing the pre-processing process;

[0033] Figure 4 A schematic diagram of phenotypic parameter calculation and mask conversion;

[0034] Figure 5 A schematic diagram of an application interface of a system. DETAILED DESCRIPTION

[0035] The following will be combined with the present invention Figures 1 to 5 , the technical solutions in the embodiments of the present invention are described in detail.

[0036] like Figure 1 The figure shows a flowchart of an image semantic segmentation method for wheat seedlings in the tillering stage. The image quality is improved by strictly controlling the time, location and environmental parameters of the acquisition. The preprocessing process is also improved to further improve the image quality and facilitate subsequent segmentation processing. Finally, the improved segmentation network model is used for training and the corrected image is used to calculate multiple phenotypic parameters, making the evaluation results more comprehensive and accurate.

[0037] The method specifically comprises the following steps:

[0038] S1. Set up multiple calibration frames in multiple areas of the wheat field, with a color calibration plate set up every 10 areas. Use a drone to collect multiple first images of the wheat seedlings, including RGB and hyperspectral images. A 0.33m x 0.33m calibration frame is set up every 5 areas of the wheat field, corresponding to the selected sampling locations.

[0039] like Figure 2 The figure below is a schematic diagram of drone-generated images. Each hyperspectral image captured by a drone includes four spectral bands: NIR, R, G, and RE. NIR stands for near-infrared, R for infrared, G for green, and RE for red-edge. These four bands are required to subsequently obtain various spectral parameters.

[0040] Before using a drone to capture multiple first images, it is necessary to select a favorable weather environment to capture the images to ensure image clarity and facilitate subsequent processing. Specifically, you can select an afternoon time period with cloud cover ≤ 8%, relative humidity ≤ 25% ≤ 35%, and visibility > 15 meters. The afternoon time period is 12:00-14:00. When using a drone for collection, it is also necessary to set the drone's flight altitude to 15 meters and the camera's lens angle to the ground to 90 degrees to capture multiple first images at the same location to avoid deviations caused by different angles.

[0041] S2. First, use the linear regression color correction algorithm to preprocess each RGB image in step S1 to obtain each second image after color correction; then use the super green segmentation algorithm to calculate each second image, retain each pixel that meets the segmentation threshold and set each pixel that does not meet the segmentation threshold to 0, to obtain each third image; continue to use the isotropic filtering algorithm to patch the edges of each third image to obtain each mask image; then use the annotation software to annotate and adjust the details of each mask image.

[0042] like Figure 3 As shown, it is a display diagram of the pre-processing process; combined with Figure 3 As shown, when performing color correction in preprocessing, the specific steps are as follows:

[0043] I) Calculate the maximum and minimum values ​​of the row and column pixel positions in each subregion where the color correction plate is located, and set the number of interpolation points to 6 and 4 in the row and column directions, respectively, to perform linear interpolation. This is used to segment the color correction plate in each RGB image into 24 color blocks, and the point with the largest pixel value in each color block is used as the pixel value representative of the corresponding color block.

[0044] II) The RGB values ​​of the 24 color blocks in each RGB image are combined using the polynomial [1, R, G, B, RG, RB, BG, RR, BB, GG] to obtain each matrix Vf; 1 is a constant term, R, G, and B correspond to the original values ​​of the red, green, and blue color channels, respectively, and RG, RB, BG, RR, BB, and GG correspond to the product of two color channels. The RGB values ​​of the 24 color blocks of the standard color chart are then combined into a matrix C of [3, 24]. An objective function is constructed to represent the difference between each matrix Vf and the corresponding matrix C. When the sum of squares of any objective function is minimized, the objective function is the corresponding optimal transformation matrix A, making each RGB image as consistent as possible with the standard color chart in RGB. The objective function can be obtained using a least squares optimization algorithm. In the matrix C of [3, 24]: the first dimension [3] indicates that each color block has three color channels: red (R), green (G), and blue (B); the second dimension

[24] indicates that there are a total of 24 different color blocks. For example, the first row will contain the red component values ​​for all 24 color patches, the second row the green component values, and the third row the blue component values.

[0045] III) using each transformation matrix A to transform each corresponding RGB image respectively to obtain each second image after color correction.

[0046] After completing the color correction, this embodiment continues to use the super green segmentation algorithm to calculate each second image, calculates the ExG value of each second image and sets the segmentation threshold to 25, regards each pixel point with ExG>25 as the pixel point where the wheat seedlings are located and retains it, and regards each pixel point with ExG≤25 as the pixel point that does not belong to the wheat seedlings and sets it to 0.

[0047] Then, the edges of each third image are repaired using an isotropic filtering algorithm. The specific steps are as follows:

[0048] 1) Traverse each non-zero pixel in the third image, use the convolution kernel of the Sobel operator for each non-zero pixel at position (i, j), and calculate the gradient amplitude of the brightness change rate in the four directions of horizontal left (i-1, j), horizontal right (i+1, j), vertical up (i, j-1), and vertical down (i, j+1) , these gradient amplitudes are also the directional derivatives in the horizontal and vertical directions respectively, and the subscripts W, E, N, and S correspond to left, right, up, and down respectively; among them, i and j are used to represent coordinates.

[0049] 2) The Perona-Malik model is used to control the anisotropy in the horizontal and vertical directions in step 1), and the diffusion coefficient λ in each of the four directions is determined. The diffusion coefficient equation in each direction is constructed and used to calculate the diffusion coefficient of each pixel. The gradients in the four directions of N, S, E, and W , and then use the edge stop function The gradient in each direction is weighted, and the weighted result in each direction is multiplied by the corresponding diffusion coefficient λ and then accumulated to obtain the update amount of each corresponding pixel point, which is used to calculate the update amount of each pixel point after diffusion. The value of and iterate, the stopping condition of the iteration is , Represents the gradient energy threshold and is set to 0.1. After edge patching, it can effectively smooth edges, reduce noise interference, and improve edge continuity and accuracy, thereby enhancing detail expression and overall recognition accuracy.

[0050] It's important to note that the Perona-Malik model is an edge detection and image smoothing method based on partial differential equations. It can reduce image noise while preserving important structural information. When repairing the edges of over-segmented wheat seedlings, the model adaptively adjusts the smoothness of the image to effectively preserve edge details while suppressing noise and irregular segmentation. This repairs inaccurate edges caused by over-segmentation or noise, improving the quality of the segmentation results and enhancing the accuracy of wheat seedling recognition.

[0051] After preprocessing, you can use the ISAT with segment anything annotation software to annotate and adjust the preprocessed mask image. The calibration frame and color correction plate are also annotated as two new categories. Detail adjustment refers to manual fine-tuning of the annotation details when the automatic annotation software does not produce satisfactory results.

[0052] S3. Perform semantic segmentation on each mask image after detail adjustment in step S2 and each first image in step S1, including: scale division, parameter adjustment, constructing a segmentation network model based on Segformer, model improvement, and model training. The model improvement includes optimizing the segmentation network model based on a spatial and channel-wise co-attention module, a cross-layer feature pyramid module, and a dilated spatial convolutional pooling pyramid module. Segformer is an advanced model architecture for semantic segmentation, particularly well-suited for dense prediction tasks on high-resolution images.

[0053] In this embodiment, the ratio division includes: taking each first image and each mask image after detail adjustment as model input, and dividing them into training set, test set and validation set in the ratio of 7:2:1. The parameter adjustment includes: using AdamW optimizer to adjust the parameters; when adjusting the parameters, the learning rate is set to 6×10 -5 , the learning decay coefficient is 0.8, the number of samples per batch iteration is 8, and the number of training rounds is 200.

[0054] like Figure 4 The figure shows a schematic diagram of phenotypic parameter calculation and mask conversion, combined with Figure 4 As shown in Figure 1, when the model inputs the RGB original image and its corresponding mask image, it enters the SCSA module and the CFPT module in sequence, and then performs the overlapping image embedding operation with a block size of 7x7, a step size of 4, and a padding of 3. Finally, it passes through the four-stage Transformer Block module, each stage adopts a resolution reduction ratio and a channel number of 、 、 as well as The decoder processes the multi-scale feature maps, each with a resolution of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image, output by the encoder. The feature channels of each stage are unified to 256 through 1x1 convolution, and then aligned and channel-wise spliced. Subsequently, multiple lightweight MLP layers are stacked, and finally the ASPP module is used. The ASPP module includes four dilated convolution functions with dilation rates of [1, 6, 9, and 12], dimension splicing, and convolution dimensionality reduction functions. The multi-scale feature maps are then fed into the ClassSeg convolution of the segmentation network model. Finally, the ClassSeg convolution generates the final segmentation mask, with the number of channels equal to the number of categories.

[0055] When improving the model, a spatial and channel collaborative attention module is added to the encoder head of the segmentation network model, so as to eliminate irrelevant background areas in spatial attention and weaken redundant channels in channel attention, reducing computational overhead; a cross-layer feature pyramid module is added after the spatial and channel collaborative attention module to integrate shallow details and deep semantics, enhance the expressive power of feature pyramids of different sizes, and improve detection accuracy; a void spatial convolution pooling pyramid is added after the multi-layer perceptron of the decoder in the segmentation network model.

[0056] The spatial and channel-wise co-attention module consists of a spatial attention branch submodule and a channel-wise attention branch submodule. In the spatial attention branch submodule, the input feature map X of shape (B, C, H, W) is first subjected to global average pooling and global max pooling along the channel dimension, generating two spatial descriptors of shape (B, C, 1, 1). These descriptors are then merged via a concatenation operation and processed through a convolutional layer and activation function, ultimately outputting normalized spatial attention weights of shape (B, 1, H, W). In the channel-wise attention branch submodule, global average pooling is used to generate channel descriptors, and a fully connected layer generates channel attention weights. These weights are then multiplied channel-wise with the input features in the input feature map in vector form, outputting normalized channel attention weights of shape (B, C, 1, 1). Finally, the mean of the normalized spatial attention weight and normalized channel attention weight tensors is calculated and used as the output of the spatial and channel-wise co-attention module.

[0057] In the cross-layer feature pyramid module, the input is first divided into 4 groups of different multi-level feature maps, each group with different spatial resolutions and different channel dimensions. Through the cross-layer feature integration submodule, the high-level features of each group of multi-level feature maps are upsampled to low-level resolution. Then, it enters the cross-layer attention submodule, which is equipped with a cross-channel attention mechanism and a spatial channel attention mechanism. In the cross-channel attention mechanism, any input feature of shape (H, W, C) is decomposed into three feature matrices including query vector, key vector and value vector through linear transformation, and then the dot product of the transposed matrix of the query vector and the key vector is calculated, that is, the dimensions of the three are all H×W×C; at the same time, through the scaling factor Normalize, d K The dimension of the value vector is , and the softmax activation function is applied to the normalized result to generate the attention weight and weighted sum it with the value vector to obtain the feature after channel attention enhancement.

[0058] Then, in the spatial channel attention mechanism, the channel-attention-enhanced features are concatenated with any of the aforementioned input features to generate spatial-channel hybrid attention features, which are used to sort the output results from high to low resolution. Finally, a new pyramid structure is constructed through lateral connections and upsampling operations. The spatial-channel hybrid attention features of this new pyramid structure are input into the shared detection head of the segmentation network model to obtain the tensor (H, W, K) for model training, where K is the number of categories.

[0059] S4. After obtaining the results of the model training in step S3, each corresponding evaluation index is calculated for each mask image after detail adjustment; the evaluation indexes include row seedling area and density.

[0060] When calculating each corresponding evaluation index for any mask image that has been adjusted in detail, first extract the red part corresponding to the calibration frame from the mask image and extract the edge contour of the correction frame; then use perspective transformation to map the pixels in the four corners of the correction frame to a new image of the same size as the mask image, and divide the number of mapped pixels by the actual area of ​​the correction frame to obtain the scale factor, the actual area is 0.33 meters × 0.33 meters, and then multiply the number of mask pixels in each row of wheat seedlings by the scale factor to obtain the row seedling area of ​​each row of wheat seedlings; then, use depth-first search-based clustering on the mask area of ​​the wheat seedlings in the mask image to calculate the minimum convex polygon of the enclosing contour of each cluster cluster, calculate the average value of the ratio of the pixels contained in all cluster clusters and the pixels contained in the minimum contour polygon, and use the average value as the density.

[0061] S5. First, estimate each transformation matrix from each spectral image to the corresponding RGB image through feature matching, and control the alignment of each spectral image with the corresponding RGB image based on each transformation matrix; continue to perform mask resampling to correct each spectral image, and then calculate the various spectral indices of the wheat seedlings, and display the various spectral indices and each evaluation indicator in step S4 as phenotypic parameters.

[0062] When performing feature matching, the four corners of the correction box in each RGB image after detail adjustment and the corresponding spectral image are first used as high-confidence matching point pairs. The RANSAC algorithm is used to estimate the transformation matrix from each spectral image to the corresponding RGB image, and the images are aligned through perspective transformation.

[0063] When performing mask resampling, each mask image is resampled according to the resolution ratio of each spectral image and each corresponding RGB image. After resampling to the same mask size in each corresponding spectral image, each resampled mask image is applied to each corresponding spectral image. The average soil partial reflectance of each pixel in the soil area in the four spectral bands of NIR, R, G, and RE is calculated, and the average value is subtracted from the original reflectance of each pixel in the four spectral wavelengths to obtain the reflectance of the wheat seedlings after eliminating the soil reflection in each spectral band. At the same time, the four spectral indices of the wheat seedlings are calculated: normalized difference vegetation index, normalized difference red edge vegetation index, chlorophyll index, and vegetation cover.

[0064] For the four spectral indices, the Normalized Difference Vegetation Index (NDVI) is calculated by taking spectral images from the near-infrared and infrared bands, calculating the quotient of the difference value (L1) and the sum value (L2) of all near-infrared and infrared spectral images with the same coordinates. The average of all these quotients is used as the estimated NDVI. The Normalized Difference Red Edge Vegetation Index (NDRI) is calculated by taking spectral images from the near-infrared and red-edge bands, calculating the quotient of the difference value (L3) and the sum value (L4) of all near-infrared and red-edge spectral images with the same coordinates. The average of all these quotients is used as the estimated NDRI. The Chlorophyll Index (CHI) is calculated by taking spectral images from the near-infrared (NIR) and green light (G) bands, calculating the average of the quotients of all near-infrared and green-edge spectral images with the same coordinates. The value obtained by subtracting one from the average is used as the estimated chlorophyll index. Vegetation coverage is calculated by subtracting the normalized difference vegetation index of the soil from the normalized vegetation index at pixel (i, j), and then dividing it by the difference between the normalized difference vegetation index of wheat seedlings and the normalized difference vegetation index of soil to obtain an estimated value of vegetation coverage.

[0065] like Figure 5 The figure shows a schematic diagram of the application interface of a system used to support the aforementioned image semantic segmentation method. The system's application interface consists of a functional section and a display section. The functional section includes a data loading module, a category adding module, an image annotation and detail fine-tuning module, a parameter calculation module, a spectral image annotation module, and a data export module. The display section displays the results of each module. The data loading module retrieves each image by selecting a specified image folder. The category adding module is used to enter a specified category name and set the annotation and mask color. The image annotation and detail fine-tuning module records and executes matrix transformations, annotates each mask image, and makes detail adjustments. The spectral image annotation module annotates spectral images, performs feature matching, and overlays each resampled mask image onto its corresponding spectral image. The parameter calculation module supports the calculation of row emergence area, compactness, normalized difference vegetation index, normalized difference red edge vegetation index, chlorophyll index, and vegetation cover. The data export module generates each mask image and its corresponding .json file, as well as the folder where the results are saved. The .json file format is commonly used for storing and transmitting data and is highly versatile.

[0066] Specifically, the functional section consists of six modules: the Data Loading module in the main toolbar, the Category Adding module in the main toolbar, the Image Annotation and Detail Adjustment module, the Spectral Image Annotation module, and the Data Export module. The display interface includes the main toolbar, the image annotation interface, the annotation category display box, the parameter display box, the mask edge fine-tuning function window, and the image list.

[0067] The data loading module opens the image files in the folder by selecting the specified image folder. The image list displays a thumbnail list of the folder images. Users can click on the image they want to operate and use the mouse wheel to zoom in and out and move the image to observe the details of the image from different dimensions.

[0068] In the Add Category module, you can enter a specific category name and set the color of the annotation mask. After clicking OK, all added categories will be displayed in the Category Display box. The number after the category name represents the number of masks.

[0069] Image annotation and detail fine-tuning module. When using the annotation software, the software has pre-loaded the weights of Mai Miao's semantic segmentation model. When the user performs operations such as scaling and moving on the annotated image, the software will record the view transformation matrix of this series of operations. When the user clicks on a pixel position of the image in the current window, the coordinates of the cropped image in the current window are converted to coordinates under the original image size through the view transformation matrix, the mask of the cropped image under the original image is calculated, and the mask polygon of the current cropping window is rendered. If the mask polygon does not meet the user's expectations, the mask can also be fine-tuned by clicking on the polygon vertex of the mask outline. In order to achieve visual synchronization, we have also added double-buffered drawing technology, which uses a double-layer area. The bottom area displays the original image, the upper area renders the mask in real time, and realizes dynamic interpolation of the mask outline to ensure smooth display of the mask at different zoom levels.

[0070] The spectral image annotation module can apply annotations to spectral images through the spectral image annotation function. Select the spectral image to be annotated through the drop-down box, and cover the upper area of ​​the spectral image through eigenvalue matching and scale transformation.

[0071] The parameter calculation module supports calculations of Emergence Area, Compactness, Normalized Difference Vegetation Index (NDVI), Normalized Difference Red Edge (NDRE), Green Wave Chlorophyll Index (CI), and Fractional Vegetation Cover (FVC). Click the desired parameter in the drop-down box, and a selection window will pop up. For example, calculating the NDVI requires images in the NIR and R channels. By selecting the near-infrared (NIR) and infrared (R) channel images in the window, the calculated NDVI parameters will be displayed in the Parameter area.

[0072] The data export module can save the generated mask image, its corresponding .json file, and the calculated phenotypic indicators to the specified folder by clicking the Save button.

[0073] In summary, the present invention first significantly improves the quality of the input image during preprocessing through color correction based on linear regression, super-green segmentation and improved anisotropic diffusion filtering, ensuring the accuracy and reliability of subsequent segmentation tasks; it also uses an improved segmentation network model and adds multiple modules for improvement to further enhance the feature extraction capability of the model; it also uses multiple transformations and feature matching to effectively eliminate background interference and calculate multiple parameters for display, making the evaluation and prediction more comprehensive; in addition, it also provides a system to support the operation of this method, which is convenient for actual operation, greatly enhances convenience, and has significant progress.

[0074] The above embodiments are only for illustrating the technical idea of ​​the present invention and cannot be used to limit the protection scope of the present invention. Any changes made on the basis of the technical solution in accordance with the technical idea proposed by the present invention shall fall within the protection scope of the present invention.

Claims

1. A semantic segmentation method for images of wheat seedlings in the tillering stage, characterized in that: The steps include: S1. Setting up multiple calibration frames in multiple areas of a wheat field, with a color calibration plate set up in every 10 areas, and using a drone to collect multiple first images of the wheat seedlings, the first images including RGB images and hyperspectral images; S2. Preprocess each RGB image in step S1 using a linear regression color correction algorithm to obtain each color-corrected second image; then calculate each second image using a super-green segmentation algorithm, retain each pixel that meets the segmentation threshold and set each pixel that does not meet the segmentation threshold to 0, to obtain each third image; continue to use an anisotropic filtering algorithm to patch the edges of each third image to obtain each mask image; then use annotation software to annotate and adjust the details of each mask image; S3. Perform semantic segmentation on each mask image after detail adjustment in step S2 and each first image in step S1, including sequentially performing scale division, parameter adjustment, constructing a segmentation network model with Segformer as the main trunk, model improvement, and model training; wherein the model improvement includes: optimizing the segmentation network model based on a spatial and channel collaborative attention module, a cross-layer feature pyramid module, and a dilated spatial convolution pooling pyramid module; when performing model improvement, adding a spatial and channel collaborative attention module to the encoder head of the segmentation network model, adding a cross-layer feature pyramid module after the spatial and channel collaborative attention module, and then adding a dilated spatial convolution pooling pyramid after the multi-layer perceptron of the decoder in the segmentation network model; S4. After obtaining the results of the model training in step S3, each evaluation index corresponding to each mask image after detail adjustment is calculated; the evaluation indexes include row seedling area and density; S5. First, estimate each transformation matrix from each spectral image to the corresponding RGB image through feature matching, and control the alignment of each spectral image with the corresponding RGB image based on each transformation matrix; continue to perform mask resampling to correct each spectral image, and then calculate the various spectral indices of the wheat seedlings, and display the various spectral indices and each evaluation indicator in step S4 as phenotypic parameters.

2. The image semantic segmentation method for wheat seedlings in the tillering stage according to claim 1, characterized in that: In step S1, a 0.33m x 0.33m calibration frame is set up for every 5 areas. Each hyperspectral image includes four spectral bands: NIR, R, G, and RE. The NIR spectral band represents the near-infrared band, the R spectral band represents the infrared band, the G spectral band represents the green band, and the RE spectral band represents the red-edge band. Before using a drone to collect multiple first images, select the afternoon time period with cloud cover ≤ 8%, relative humidity ≤ 25% ≤ 35%, and visibility > 15 meters for collection. The afternoon time period is 12:00-14:00; when using a drone for collection, set the drone's flight altitude to 15 meters and the camera's lens angle to the ground to 90 degrees.

3. The image semantic segmentation method for wheat seedlings in the tillering stage according to claim 1, characterized in that: The steps for color correction in step S2 are as follows: I) calculating the maximum and minimum values ​​of the row and column pixel positions in each sub-region where the color correction plate is located, and setting the number of interpolation points to 6 and 4 in the row and column directions, respectively, to perform linear interpolation to segment the color correction plate in each RGB image into 24 color blocks, and taking the point with the largest pixel value in each color block as the pixel value representative of the corresponding color block; II) The RGB values ​​of the 24 color blocks in each RGB image are combined into a [1, R, G, B, RG, RB, BG, RR, BB, GG] polynomial combination to obtain each matrix Vf; the RGB values ​​of the 24 color blocks of the standard color chart are then combined into a [3, 24] matrix C, and an objective function is constructed to represent the difference between each matrix Vf and matrix C. When the sum of squares of the objective function is minimized, the optimal transformation matrix A is obtained; where 1 is a constant term, R, G, and B correspond to the original values ​​of the red, green, and blue color channels, respectively, and RG, RB, BG, RR, BB, and GG correspond to the product of two color channels, respectively; III) using each transformation matrix A to transform each corresponding RGB image respectively to obtain each second image after color correction.

4. The image semantic segmentation method for wheat seedlings in the tillering stage according to claim 1, characterized in that: When the super green segmentation algorithm is used to calculate each second image in step S2, the super green index ExG value of each second image is calculated and the segmentation threshold is set to 25. Each pixel point with a super green index ExG value > 25 is regarded as a pixel point where the wheat seedlings are located and is retained. Each pixel point with a super green index ExG value ≤ 25 is regarded as a pixel point not belonging to the wheat seedlings and is set to 0.

5. The image semantic segmentation method for wheat seedlings in the tillering stage according to claim 1, characterized in that: In step S2, the steps of using an anisotropic filtering algorithm to repair the edge of any third image are as follows: 1) Traverse each non-zero pixel in the third image, use the convolution kernel of the Sobel operator for each non-zero pixel at position (i, j), and calculate the gradient amplitude of the brightness change rate in the four directions of horizontal left (i-1, j), horizontal right (i+1, j), vertical up (i, j-1), and vertical down (i, j+1) 、 、 、 , subscripts W, E, N, and S correspond to left, right, top, and bottom respectively; where i and j are used to represent coordinates; 2) The Perona-Malik model is used to control the anisotropy in the horizontal and vertical directions in step 1), and the diffusion coefficient λ in each of the four directions is determined. The diffusion coefficient equation in each direction is constructed and used to calculate the diffusion coefficient of each pixel. The gradients in the four directions of N, S, E, and W ， Then use the edge stop function The gradient in each direction is weighted, and the weighted result in each direction is multiplied by the corresponding diffusion coefficient λ and then accumulated to obtain the update amount of each corresponding pixel point, which is used to calculate the update amount of each pixel point after diffusion. The value of and iterate, the stopping condition of the iteration is , represents the gradient energy threshold and is set to 0.

1.

6. The image semantic segmentation method for wheat seedlings in the tillering stage according to claim 1, characterized in that: In step S3, the proportional division includes: taking each first image and each mask image after detail adjustment as model input, and dividing them into a training set, a test set, and a validation set in a ratio of 7:2:1; Parameter adjustment includes: using AdamW optimizer to adjust parameters; when adjusting parameters, setting the learning rate to 6×10 -5 , the learning decay coefficient is 0.8, the number of samples per batch iteration is 8, and the number of training rounds is 200 rounds.

7. The image semantic segmentation method for wheat seedlings in the tillering stage according to claim 1, characterized in that: The spatial and channel collaborative attention module includes: a spatial attention branch submodule and a channel attention branch submodule; in the spatial attention branch submodule, global average pooling and global maximum pooling are first performed on the input feature map along the channel dimension to generate two spatial descriptors; the two descriptors are then merged through a splicing operation, processed through a convolutional layer and an activation function, and finally the normalized spatial attention weights are output; in the channel attention branch submodule, global average pooling is used to generate channel descriptors and a fully connected layer is used to generate channel attention weights. The weights are multiplied channel by channel in the input feature map in vector form to output the normalized channel attention weights; the average of the normalized spatial attention weights and the normalized channel attention weights is calculated as the output; In the cross-layer feature pyramid module, the input is first divided into four groups of different multi-level feature maps, which are horizontally connected through the cross-layer feature integration submodule to upsample the high-level features of each group of multi-level feature maps to a low-level resolution. Subsequently, the module enters the cross-layer attention submodule, which is equipped with a cross-channel attention mechanism and a spatial channel attention mechanism. In the cross-channel attention mechanism, any input feature is decomposed into three feature matrices including a query vector, a key vector, and a value vector through a linear transformation. The dot product of the transposed matrix of the query vector and the key vector is then calculated and normalized by a scaling factor. The softmax activation function is applied to the normalized result to generate attention weights and the weighted summation is performed with the value vector to obtain the channel-attention-enhanced features. In the spatial channel attention mechanism, the channel-attention-enhanced features are concatenated with the arbitrary input features to generate spatial-channel hybrid attention features, which are used to arrange the output results from high to low resolution. Finally, a new pyramid structure is constructed through horizontal connection and upsampling operations, and the spatial-channel hybrid attention features of the new pyramid structure are input into the shared detection head of the segmentation network model to obtain the tensor for model training. The dilated spatial convolution pooling pyramid includes four dilated convolutions, which are used for convolution dimensionality reduction and output of multi-scale feature maps to the Class Seg convolution of the segmentation network model. The Class Seg convolution is used to generate segmentation masks during model training.

8. The image semantic segmentation method for wheat seedlings in the tillering stage according to claim 1, characterized in that: In step S4, when calculating each corresponding evaluation index for any mask image that has been adjusted for details, first extract the red part corresponding to the calibration frame from the mask image and the edge contour of the correction frame; then use perspective transformation to map the pixels in the four corners of the correction frame to a new image of the same size as the mask image, divide the number of mapped pixels by the actual area of ​​the correction frame to obtain a scale factor, and then multiply the number of mask pixels in each row of wheat seedlings by the scale factor to obtain the row seedling area of ​​each row of wheat seedlings; then, use depth-first search-based clustering for the mask area of ​​the wheat seedlings in the mask image, calculate the minimum convex polygon of the enclosing contour of each cluster cluster, calculate the average value of the ratio of the pixels contained in all cluster clusters and the pixels contained in the minimum contour polygon, and use the average value as the density.

9. The image semantic segmentation method for wheat seedlings in the tillering stage according to claim 2, characterized in that: In step S5, when performing feature matching, the four corners of the correction frame in each RGB image after detail adjustment and the corresponding spectral image are first used as high-confidence matching point pairs. The RANSAC algorithm is used to estimate the transformation matrix from each spectral image to the corresponding RGB image, and the images are aligned through perspective transformation. When performing mask resampling, each mask image is resampled according to the resolution ratio of each spectral image and each corresponding RGB image. After resampling to the same mask size in each corresponding spectral image, each resampled mask image is applied to each corresponding spectral image. The average soil partial reflectance of each pixel in the soil area in the four spectral bands of NIR, R, G, and RE is calculated, and the average value is subtracted from the original reflectance of each pixel in the four spectral wavelengths to obtain the reflectance of the wheat seedlings after eliminating the soil reflection in each spectral band. At the same time, the four spectral indices of the wheat seedlings are calculated: normalized difference vegetation index, normalized difference red edge vegetation index, chlorophyll index, and vegetation cover.

10. A system operating using the image semantic segmentation method for wheat seedlings in the tillering stage according to any one of claims 1 to 9, characterized in that: The application interface of the system includes a functional part and a display part. The functional part includes a data loading module, a category adding module, an image annotation and detail fine-tuning module, a parameter calculation module, a spectral image annotation module and a data export module. The display part is used to display the results of each module. Among them, the data loading module is used to obtain each image by selecting a specified image folder; Add a category module to enter the specified category name and set the label and mask colors; The image annotation and detail fine-tuning module is used to record and execute the transformation operation of each matrix, annotate each mask image, and make detail adjustments; The spectral image annotation module is used to perform spectral image annotation, feature matching, and overlay each resampled mask image onto each corresponding spectral image; The parameter calculation module is used to support the calculation of row emergence area, compactness, normalized difference vegetation index, normalized difference red edge vegetation index, chlorophyll index and vegetation cover; The data export module is used to generate each mask image and each corresponding .json file as well as the folder location where the results are saved.

Images