A wheat seedling leaf tip detection and counting method based on multi-scale feature fusion

Abstract

The application discloses a kind of field wheat seedling leaf tip detection counting method based on multi-scale feature fusion, belong to agricultural informatization and machine vision technical field.For wheat leaf mutual occlusion under field environment, background is complex and the scale difference of target is big, very small target is easy to miss detection and other problems caused by multi-platform acquisition, the application constructs wheat leaf tip dataset by smart phone and unmanned aerial vehicle multi-platform acquisition image;Subsequently, a TIPS-DETR wheat seedling leaf tip detection counting model is constructed: a double-branch fusion backbone is introduced in the feature extraction stage, a small target enhanced feature pyramid is constructed in the feature fusion stage, and an optimized loss function is introduced in the prediction stage. The model has a precision of 0.8123, an average precision (mAP 50 ) of 0.7876, a parameter reduction of 38.8%, and a calculation reduction of 35.3%. This method can effectively process multi-scale images, overcome environmental disturbances such as field weeds, straw and light changes, and achieve accurate detection and counting of wheat seedling leaf tips in field scenes while reducing the computational burden of the model.

Description

Technical Field

[0001] This invention relates to the field of wheat seedling quantity assessment technology, specifically to a field wheat seedling leaf tip detection and counting method based on multi-scale feature fusion. Background Technology

[0002] Wheat is an important food crop in my country, and its early growth status directly affects its final yield and quality. Accurately acquiring phenotypic parameters such as seedling number, leaf age, and evenness of population distribution during the wheat seedling stage is crucial for evaluating the quality of strong seedlings before winter, guiding precise variable-rate fertilization in the field, and conducting early yield measurements. Traditional agricultural methods for statistically analyzing these parameters are typically time-consuming, labor-intensive, costly, and inefficient. Remote sensing technology, however, offers real-time, efficient, and non-destructive monitoring capabilities. Therefore, utilizing machine vision to quickly and accurately acquire crop phenotypic information is particularly important in modern agriculture. Field applications of different acquisition methods also demonstrate that image recognition methods are low-cost, highly efficient, and require less skilled personnel, making them more suitable for widespread application.

[0003] Because the lower leaves of wheat plants are often overlapping and disordered in field environments, traditional whole-plant or whole-leaf segmentation counting methods are ineffective. However, each wheat leaf corresponds to only one leaf tip. Utilizing the leaf tip as a feature for identification and counting effectively avoids the problem of leaf overlap and has become a highly promising research direction for high-throughput monitoring of wheat populations. Therefore, this invention proposes a method capable of accurately extracting and fusing minute target features in complex field environments (weeds, straw, light variations), adapting to the significant scale differences brought about by multiple acquisition platforms, and reducing the computational burden on the model. This method aims to achieve high-precision detection and counting of wheat seedling leaf tips in the field, solving a key technical challenge in the current field of agricultural informatization and smart agriculture. Summary of the Invention

[0004] To address the aforementioned technical challenges, this invention proposes a method for detecting and counting wheat seedling leaf tips in the field based on multi-scale feature fusion. This invention can adapt to the significant scale differences arising from multiple acquisition platforms in complex field environments, and accurately extracts and fuses small target features while reducing the computational burden on the model, thereby achieving the detection and counting of wheat seedling leaf tips in the field.

[0005] To achieve the above objectives, this invention employs a field wheat seedling leaf tip detection and counting method based on multi-scale feature fusion, comprising the following steps:

[0006] S1: Use smartphones and drones to collect visible light images of wheat seedlings, then label the collected images using labelImg to construct an image dataset. The image dataset is then randomly divided into training, validation, and test sets in an 8:1:1 ratio, as follows:

[0007] S11: Image acquisition. After wheat sowing, image data acquisition of wheat seedlings will be carried out using smartphones and drones during the seedling emergence to four-leaf stage.

[0008] S12: Image cropping. The image acquired by the drone is cropped into blocks. The number of blocks N is determined by the following formula:

[0009]

[0010] Where W and H are the image width and height resolution, respectively, and S is the step size.

[0011] S13: Image annotation. Use the annotation tool labelImg to create rectangular annotations for the wheat seedling leaf tips in the image. The annotation strategy is to use the area above the vertex of the leaf tip as one corner of the rectangle. The diagonal of the rectangle coincides with or is parallel to the central axis of the leaf, extending along the leaf direction. The length of the diagonal (i.e., the longitudinal span of the leaf tip area) is 1–2 cm. The length of the shorter side of the rectangle is adaptively determined based on the actual width of the leaf tip to ensure complete coverage of the leaf tip outline.

[0012] S14: Dataset Construction. After data cleaning, the cropped and labeled images are randomly divided into training set, validation set, and test set 1. In addition, three challenging test sets are manually selected and constructed: Test set 2 contains challenging data from complex field environments, including images of straw, weeds, straw + weeds, and images after weeding; Test set 3 contains challenging data from different lighting and soil conditions in the field, including images of sunny / cloudy days and different soil backgrounds; Test set 4 contains data from different growth stages of wheat, including images from the one-leaf stage to the four-leaf stage.

[0013] S2: Construct the TIPS-DETR framework, a real-time object detection model based on Transformer, including a feature extraction network, a feature fusion network, and a prediction network. Specifically:

[0014] ① The feature extraction network is a dual-branch fusion backbone, MultiBackbone. First, the input raw image is fed into the HGStem layer for preliminary feature mapping and basic downsampling. Then, the preliminary features enter the heterogeneous feature extraction branch, extracting high-resolution spatial and semantic information respectively, and directly outputting small target information features. Finally, the features flow through multiple consecutively stacked Spatial-Semantic Adaptive Fusion Modules (SSAFM) for deep multi-scale feature mining and downsampling, ultimately outputting four layers of features at different resolutions. Specifically:

[0015] S211: To avoid consuming excessive computational resources on shallow feature maps, the designed feature extraction network shares information from an HGStem layer. HGStem is the initial feature extraction module in the HGNet (High Performance GPU Network). This design helps avoid excessive computation and inference time. The input image is downsampled by a convolutional module and then split into two paths. One path extracts local features through consecutive convolutional modules, while the other path uses max pooling to preserve the global signal. The two feature paths are concatenated and then output through a secondary downsampling module.

[0016] S212: After the initial mapping by HGStem, the feature map is simultaneously fed into a parallel C2f (Cross-Stage Partial Feature Fusion) module and an HGBlock (Hierarchical Graph Block) module to obtain the features and high-resolution feature layers of small targets that are passed down to the downstream deep network, denoted as S2.

[0017] S213: Regarding the feature extraction of the S2 layer, this model adopts a unique heterogeneous routing strategy: the direct output of the S2 layer comes only from the fused output of the C2f branch, and the features output by the HGBlock module are only transmitted to the downstream deep network.

[0018] S214: Subsequently, the network consists of multiple SSAFM modules stacked consecutively. Each SSAFM module contains two parallel heterogeneous paths: the first path is composed of C2f concatenated with regular convolutions, and the second path is composed of HGBlock alternating with depthwise separable convolutions. The features of the two paths are aligned and fused at the end of the module using a Multi-Scale Adaptive Spatial Attention Gate (MASAG) mechanism. The MASAG processing flow includes a multi-scale feature fusion stage, a spatial selection stage using Softmax to calculate weights, a spatial interaction stage using Sigmoid for interactive modulation, and a recalibration stage using pointwise convolutions to generate attention maps for recalibration. Through the layer-by-layer processing of multiple SSAFM modules, four layers of features at different resolutions are finally output together with the high-resolution feature layer for small objects, denoted as [S2, S3, S4, S5].

[0019] ② The feature fusion network described is the Small Target Enhanced Feature Pyramid (SOEP). In wheat field images, seedling leaf tips are typical small targets. Traditional feature pyramid networks (such as FPN or PANet) typically rely only on layers S3 to S5 for feature fusion. However, after multiple step-downsampling operations, the spatial information of small targets is almost completely lost in the deep feature map, leading to severe missed detections and underestimation of counts. This model designs the Small Target Enhanced Feature Pyramid (SOEP), which uses the S2 feature layer to fuse features rich in small target information with S3 through Space-to-Depth Convolution (SPDConv). Then, it uses the Cross-Stage Partial (CSP) concept and an improved Omni-Kernel Module (OKM) to obtain the Cross-Stage Partial Omni-Kernel Module (CSP-OKM) for feature integration. While significantly reducing the number of parameters and computational cost, SOEP achieves lossless transfer of multi-scale features and full-dimensional feature alignment. Specifically:

[0020] S221: To preserve the small target information introduced by the S2 layer, SOEP uses SPDConv instead of the traditional downsampling operator. After passing through SPDConv, the S2 feature layer is spliced ​​along the channel axis and compressed to the same scale as the S3 layer. Then, it is spliced ​​and fused with the S3 layer features to output feature F3.

[0021] S222: Flatten the feature set output by S5, and output the feature set as F5 through an attention-based intra-scale feature interaction module (containing a feedforward neural network and a multi-head attention mechanism). The specific formula is as follows:

[0022]

[0023]

[0024] in For flattening operation, As a multi-head attention mechanism, It is a feedforward neural network. This means restoring the shape of the feature to be the same as S5.

[0025] S223: The CSP-OKM module is introduced at the feature intersection node. This module is obtained by using the idea of ​​cross-stage local networks and improving upon the full-core module. It captures the discriminative information of small targets to the maximum extent with extremely low computational cost when fusing cross-layer features.

[0026] S224: After generating enhanced features F3 and F5, SOEP enters the full-scale cross-layer feature fusion stage. Using several 1x1 convolutions to uniformly map the number of channels of features [F3, S4, F5] to 256 dimensions, it enters a bidirectional weighted path containing a Reparameterization Convolution (RepC3) module and a CSP-OKM module. Top-down path: F5 is upsampled and concatenated with S4, then fed into the RepC3 module to generate transition feature P4'; P4' is upsampled and concatenated with F3, then fused by the CSP-OKM and RepC3 modules to output high-resolution prediction feature P3. Bottom-up path: P3 is downsampled using SPDConv and concatenated with P4', then fused by RepC3 to generate medium-resolution prediction feature P4; P4 is downsampled using SPDConv and concatenated with F5, then fused by the RepC3 module to output low-resolution prediction feature P5. Finally, the network outputs a set of multi-scale predicted feature representations [P3, P4, P5], which is then fed into subsequent prediction networks.

[0027] ③ The prediction network employs a visual transformation decoder and decoupled head structure. Feature sets [P3, P4, P5] at different spatial scales are input to the visual transformation decoder, and iterative correction is performed through multi-head self-attention, cross-attention, and feedforward mapping networks, combined with two-dimensional position encoding and residual connections. The decoded and refined object query sequence is introduced into the decoupled prediction head, and mapping and parsing are performed through mutually independent parallel classification and regression branches. The bounding box regression branch is calculated based on Shape-IoU, and its mathematical expression is as follows:

[0028]

[0029]

[0030]

[0031]

[0032] in, and These represent the predicted bounding box and the ground truth bounding box, respectively. and These represent the center coordinates of the predicted bounding box and the ground truth bounding box, respectively. This is the diagonal length of the smallest bounding closure region that simultaneously contains both the predicted and ground truth bounding boxes. In the item, and These are adaptive weighting coefficients introduced to address changes in the target scale. In the item, It measures the relative deviation between the predicted bounding box and the ground truth bounding box in the width and height dimensions. The sensitivity to shape penalty is set to 4 in this network.

[0033] S3: Train, validate, and test the Transformer-based real-time object detection model TIPS-DETR. Train the TIPS-DETR model using the training set, employing the AdamW optimizer with a cosine annealing learning rate strategy for backpropagation, iterating 300 times. Evaluate the mAP value on the validation set in each iteration, save the optimal number of iterations, and calculate model parameters and running speed. Train mainstream models YOLOv5S, YOLOv5M, YOLOv8S, YOLOv8M, YOLOv10M, and RT-DETR using the same data, and compare the detection accuracy and counting accuracy of the mainstream models and the TIPS-DETR model to evaluate the state-of-the-art performance of the TIPS-DETR model. Based on the RT-DETR model, progressively add improved modules to form Model 1 (RT-DETR), Model 2 (Model 1 + dual-branch fusion backbone), Model 3 (Model 2 + small target enhancement feature pyramid), and Model 4 (Model 3 + optimized loss function, i.e., TIPS-DETR), to verify the impact of each module on the wheat seedling leaf tip detection model. The detection and counting accuracy of the TIPS-DETR model were tested using a test set and challenging data to evaluate the model's effectiveness.

[0034] S4: The TIPS-DETR real-time target detection model based on Transformer was used to collect visible light images of wheat seedlings using smartphones and drones. After preprocessing according to S1, the images were input into the TIPS-DETR real-time target detection model based on Transformer to obtain the leaf tip detection results of wheat seedlings in the field and count the number of targets.

[0035] Compared with existing technologies, this invention has the following innovations: 1. This invention proposes a field wheat seedling leaf tip detection and counting method based on multi-scale feature fusion, which can detect and count wheat seedling leaf tips in different field application scenarios. This method has a wide range of applications and significantly improves detection and counting efficiency. 2. A field wheat seedling leaf tip detection network model based on multi-scale feature fusion is constructed. In the feature extraction stage, a dual-branch fusion backbone (MultiBackbone) is introduced to extract spatial details and global semantics; in the feature fusion stage, a small target enhanced feature pyramid (SOEP) is introduced to achieve lossless cross-layer aggregation of small target features; in the prediction stage, an optimized loss function (Shape-IoU) is introduced to improve regression accuracy by utilizing shape constraints. This network can effectively process wheat seedling images of different scales under multiple acquisition platforms, adapt to the complex environment of the field, and overcome interference problems such as weeds, straw backgrounds, and changes in light intensity, achieving the detection and counting of small target leaf tips with low computational burden. Attached Figure Description

[0036] Figure 1 This describes a field wheat seedling leaf tip detection and counting method based on multi-scale feature fusion.

[0037] Figure 2 This diagram illustrates the steps of a method for identifying and counting the leaf tips of wheat seedlings.

[0038] Figure 3 This represents the experimental counting results compared with mainstream models.

[0039] Figure 4 This represents the test results for data with different levels of difficulty.

[0040] Figure 5 This section presents the visualization results of different evaluation indicators of the comparative model at different growth stages of wheat.

[0041] Figure 6 The representation method shows the visualization results on smartphones and drones in a field setting. Detailed Implementation

[0042] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0043] like Figure 1 and Figure 2 As shown, the present invention specifically includes the following steps:

[0044] S1: Visible light images of wheat seedlings were collected using smartphones and drones. The collected images were then labeled using labelImg to construct an image dataset, which included 3488 training images, 436 validation images, and 436 test images. In addition, 1200 images with a certain level of detection difficulty were manually selected for labeling, and three challenging test sets were constructed: Test set 2 contained challenging data from complex field environments, including 400 images of straw, weeds, straw + weeds, and images after weeding; Test set 3 contained challenging data from different lighting and soil conditions in the field, including 400 images of sunny / cloudy days and different soil backgrounds; Test set 4 contained data from different wheat growth stages, including 400 images from the one-leaf to four-leaf stage. See Table 1 for the specific datasets.

[0045] Table 1 Dataset Construction

[0046]

[0047] S2: Construct the TIPS-DETR framework, a real-time object detection model based on Transformer, including a feature extraction network, a feature fusion network, and a prediction network. Specifically:

[0048] ① The feature extraction network is a dual-branch fusion backbone, MultiBackbone. First, the input raw image is fed into the HGStem layer for preliminary feature mapping and basic downsampling. Then, the preliminary features enter the heterogeneous feature extraction branch, extracting high-resolution spatial and semantic information respectively, and directly outputting small target information features. Finally, the features flow through multiple consecutively stacked spatial-semantic adaptive fusion modules (SSAFM) for deep multi-scale feature mining and downsampling, ultimately outputting four layers of features at different resolutions. Details are as follows:

[0049] S211: To avoid consuming excessive computational resources on shallow feature maps, the designed feature extraction network shares information from an HGStem layer, which is the initial feature extraction module in the HGNet network. This design helps avoid excessive computation and inference time. This layer consists of a lightweight contiguous convolutional module (ConvBNAct), namely a concatenated convolutional kernel, a Batch Normalization (BN) module, and an activation function. After downsampling through a ConvBNAct module, HGStem divides the input image feature map into two parts for local processing. One part uses a contiguous ConvBNAct module to finely extract high-frequency local features, while the other part uses a parameterless MaxPool to retain low-frequency global saliency signals. Subsequently, the two feature paths are concatenated and fused along the channel dimension and then downsampled again through a ConvBNAct module. This allows for rapid reduction of feature map resolution while efficiently and losslessly preserving rich multi-layered low-level visual information.

[0050] S212: After initial mapping by HGStem, the feature maps are simultaneously fed into a parallel C2f module and an HGBlock module, resulting in features and a high-resolution feature layer for small targets to be passed down to the downstream deep network, denoted as S2. In the C2f module, the input features first pass through a 1*1 ConvBNAct module and then split into two paths: one path directly skips and retains the data, while the other path sequentially passes through multiple Bottleneck modules with their own residual structures, i.e., two cascaded 3*3 ConvBNAct modules extract deep information. Subsequently, the skip branch and the cascaded output of each Bottleneck are concatenated through multiple channels. Finally, a 1*1 ConvBNAct module is used for fusion, compression, and output. In HGBlock, the input features are first extracted by multiple parallel ConvBNAct modules, and then concatenated with the original input channels. Next, the ESC module performs attention-weighted purification on the channels. The ESC module uses 1*1 convolution to convert the number of channels to the number of channels to be output, and then calculates the mean of each layer's features. After calculating the Sigmoid function of each layer's mean, each layer's features are multiplied by the corresponding Sigmoid result. Finally, the original input is fused again through residual connections to complete the output.

[0051] S213: Regarding feature extraction from layer S2, this model employs a unique heterogeneous routing strategy: the direct output of layer S2 originates solely from the C2f branch. This is because the C2f module, leveraging its rich cross-stage gradient flow and multi-path concatenation mechanism, can perform high-fidelity extraction of fine-grained geometric edges. Simultaneously, the parallel HGBlock module does not participate in the direct output of layer S2 but is dedicated to transmitting features to downstream deeper networks. Due to the smaller receptive field of shallow networks, prematurely outputting HGBlock features can easily lead to semantic dilution of local spatial information.

[0052] S214: Subsequently, the network consists of multiple SSAFM modules stacked consecutively. Each SSAFM module incorporates two parallel heterogeneous feature processing paths: one composed of C2f and conventional convolutions in series. Due to its physically-level lossless skip connections and strong binding to local pixels, it retains rich spatial geometric features and fine edge information, continuously tracking and enhancing high-resolution local details. The other path alternates between HGBlock and depthwise separable convolutions. Its active dimensionality reduction of spatial location and enhanced cross-channel feature recombination effectively filters out spatial background redundancy and highly purifies the class features of the core target. Furthermore, it utilizes channel decoupling and hierarchical aggregation mechanisms to abstract features from pixel-level to high-level contextual semantics. To break down the feature barriers between these two heterogeneous paths and avoid differences between features from different sources during information fusion, a multi-scale adaptive spatial attention gate mechanism (MASAG) is introduced at the end of the module. MASAG dynamically adjusts the spatial receptive field to highlight spatially relevant features and suppress background interference. This mechanism deeply aligns the spatial details extracted by the C2f branch with the global semantics extracted by the HGBlock branch. Its processing flow comprises four core stages. The first stage is multi-scale feature fusion, which expands the receptive field of local spatial features through deep convolution and dilated convolution, while simultaneously extracting extensive contextual information from semantic features using channel-level pooling (average pooling and max pooling). The two are integrated to form a comprehensive feature map containing multi-scale information. The second stage is spatial selection, which maps the fused feature map to the two channels and calculates spatial selectivity weights using channel-level softmax. This step dynamically adjusts the receptive field, enhancing attention to key regions (such as leaf tip edges) while suppressing irrelevant background noise. The third stage is spatial interaction and cross-modulation. In this stage, local details are enhanced with global context through Sigmoid spatial weights, while global features are also integrated with high-resolution local details. Element-wise multiplication ensures that the output features perfectly blend local accuracy and macroscopic structural cognition. The fourth stage is recalibration, where the interacted features are used to generate a highly focused attention map through pointwise convolution and Sigmoid activation. This attention map is used to multiply and recalibrate the spatial detail features of the initial input, ensuring that the features passed to the next deep network have an extremely accurate and context-aware spatial representation. Through layer-by-layer processing by multiple SSAFM modules, it finally outputs four layers of features at different resolutions together with the high-resolution feature layer for small objects, denoted as [S2, S3, S4, S5], achieving deep coupling between local fine-grained features and global macro-semantics.

[0053] ② The feature fusion network described is the Small Target Enhanced Feature Pyramid (SOEP). In wheat field images, seedling leaf tips are typical small targets. Traditional feature pyramid networks (such as FPN or PANet) typically rely only on layers S3 to S5 for feature fusion. However, after multiple step-downsampling steps, the spatial information of small targets is almost completely lost in the deep feature map, leading to severe missed detections and underestimation of counts. This model designs the Small Target Enhanced Feature Pyramid (SOEP), which uses the S2 feature layer to fuse features rich in small target information with S3 via SPDConv. Then, it uses the CSP concept and an improved CSP-OKM module for feature integration. While significantly reducing the number of parameters and computational cost, SOEP achieves lossless transfer of multi-scale features and full-dimensional feature alignment. Specifically:

[0054] S221: During the top-down and bottom-up cross-layer fusion process of the feature pyramid, conventional convolution or pooling operations can lead to the irreversible loss of fine-grained features. To protect the small target information introduced by the S2 layer, SOEP uses SPDConv instead of the traditional downsampling operator. SPDConv divides the high-resolution feature map into multiple complementary sub-feature maps through a straddle slicing sampling mechanism. These sub-feature maps are then concatenated and merged along the channel axis, thereby losslessly reshaping and interleaving the feature blocks in the spatial dimension to the channel dimension. This structural design effectively reduces the spatial resolution of the feature map while completely avoiding pixel-level loss due to straddle slippage, ensuring that small target features are not destructively erased by pooling operations when propagating within the pyramid. After passing through SPDConv, the fine-grained information in the spatial dimension of the S2 feature layer is interleaved and compressed to the same scale as the S3 layer. Subsequently, the downsampled S2 layer features are concatenated and fused with the S3 layer features, and the output feature is denoted as F3.

[0055] S222: Flatten the feature set output by S5, and output the feature set as F5 through an attention-based intra-scale feature interaction module (containing a feedforward neural network and a multi-head attention mechanism). The specific formula is as follows:

[0056]

[0057]

[0058] in For flattening operation, As a multi-head attention mechanism, It is a feedforward neural network. This means restoring the shape of the feature to be the same as S5.

[0059] S223: In field scenarios, wheat seedling leaf tips have a small pixel ratio and exhibit drastic scale variations across different acquisition platforms. To maximize the capture of discriminative information for small targets during cross-layer feature fusion, this model introduces an OKM module at the feature intersection node. After the input features undergo an initial 1*1 convolution, the OKM module branches them into three parallel receptive field branches to adaptively learn multi-dimensional feature representations from local to global. The three branches are: a local branch using 1*1 depthwise convolutions; a large branch using a combination of depthwise convolutions with ultra-large kernels; a large branch using 63*1 and 1*63 one-dimensional strip convolutions and 63*63 two-dimensional large kernel convolutions in parallel; and a global branch that connects the attention module. The outputs of these three branches are then element-wise added to the residual connections of the input features and fused through a final 1*1 convolution. However, the standard OKM module, due to its large combination of large kernel convolutions and dense parallel branch operations, leads to a sharp increase in the number of network parameters and computational complexity. To overcome this computational bottleneck, this model refactors the standard OKM module using the CSP (Convolutional Programming) approach, resulting in a CSP-OKM module with extremely low computational cost and superior feature preservation. Upon receiving input features, the CSP-OKM module first processes them using standard convolutions. Then, it splits the features in two along the channel dimension: one part is directly mapped to the tail, preserving the original gradient flow and shallow spatial details; the other part is fed into the standard OKM module for deep, multi-scale feature mining. Finally, these two sets of features are concatenated and merged along the channel dimension, and then fused through a terminal convolution. The introduction of the CSP architecture not only reduces the computational burden by more than half but also provides richer gradient combinations through the segmentation and concatenation mechanism. At the critical node of cross-layer feature fusion, the CSP-OKM module achieves multi-scale adaptive feature integration of microscopic details and macroscopic semantics with extremely low computational cost.

[0060] S224: After generating enhanced features F3 and F5, SOEP enters the full-scale cross-layer feature fusion stage. The network first uses several 1*1 convolutions to uniformly map the number of channels of features [F3, S4, F5] to 256 dimensions, forming a standardized feature set with different scales but aligned channels. Subsequently, these features are aggregated across layers through a bidirectional weighted path from top to bottom and bottom to top. To ensure efficient fusion after multi-scale feature concatenation, a RepC3 module is introduced after each concatenation operation. The RepC3 module combines the CSP mechanism with structural reparameterization technology. During the training phase, this module adopts a multi-branch topology to obtain rich gradient information and stronger feature representation capabilities; while during the inference phase, its internal multi-branch structure can be mathematically equivalent and folded into a single 3*3 convolution operator, thereby reducing computation and improving inference speed. Top-down path: After upsampling, F5 is concatenated with S4 in the channel dimension, and the concatenated feature is fed into the RepC3 module to generate transition feature P4'. Subsequently, P4' is upsampled again and concatenated with F3. After adaptive alignment by the CSP-OKM module, it is fused with the RepC3 module to output the final high-resolution prediction feature P3. Bottom-up path: P3 is losslessly downsampled using SPDConv, concatenated with P4', and after passing through the RepC3 module, the final medium-resolution prediction feature P4 is generated. Then, P4 is downsampled again using SPDConv and concatenated with F5. Similarly, it is fused by the RepC3 module to output the final low-resolution prediction feature P5. Finally, the network outputs a multi-scale prediction feature representation set [P3, P4, P5], which is input into subsequent prediction networks, providing feature support for leaf tip bounding box decoding in complex backgrounds.

[0061] ③ The prediction network adopts a visual transformation decoding and decoupling head structure. The feature sets [P3, P4, P5] of different spatial scales output by the SOEP module are directly input into the visual transformation decoder. This decoding network adopts a layered architecture, and its core computational units include multi-head self-attention, cross-attention, and feedforward mapping networks. During feature transfer, the self-attention mechanism mainly acts within the object query sequence, suppressing redundant repetitive predictions by calculating the interdependence between leaf tip targets; the cross-attention module is responsible for guiding the query vector and multi-scale image features to perform deep aggregation, adaptively extracting discriminative information of the leaf tip. To ensure complete perception of the global spatial context, two-dimensional position encoding is superimposed on the feature input, supplemented by residual connections to ensure smooth transmission of deep gradients. As the decoding layer deepens, the query vector undergoes recursive iteration and correction. After decoding and refinement, the feature sequence rich in target information is introduced into the decoupling prediction head. The decoupling head parses the target by constructing two independent parallel mapping branches, and finally outputs the class confidence score of wheat seedling leaf tips and the target bounding box coordinates. In a field setting, the tips of wheat seedling leaves in an image are slender and extremely small targets. Traditional regression loss functions (such as the IoU series), based on the intersection-union ratio (IoU), are highly sensitive to pixel-level shifts in small targets, making it difficult for predicted bounding boxes to accurately fit the true geometric contours of the leaf tips. This model introduces the Shape-IoU loss function for depth optimization in the bounding box regression branch. Unlike traditional methods that only focus on overlap and center point distance, Shape-IoU directly incorporates the shape and scale attributes of the target itself into the regression penalty term, thus compensating for the mathematical deficiencies of traditional metrics for small targets. Specifically, Shape-IoU focuses on the aspect ratio of the bounding box, dynamically calculating the scale deviation between the predicted and ground truth bounding boxes in the width and height dimensions. Its mathematical expression is as follows:

[0062]

[0063]

[0064]

[0065]

[0066] in, and These represent the predicted bounding box and the ground truth bounding box, respectively. and These represent the center coordinates of the predicted bounding box and the ground truth bounding box, respectively. This is the diagonal length of the smallest bounding closure region that simultaneously contains both the predicted and ground truth bounding boxes. In the item, and These are adaptive weighting coefficients introduced to address changes in the target scale. In the item, It measures the relative deviation between the predicted bounding box and the ground truth bounding box in the width and height dimensions. The sensitivity to shape penalty is set to 4 in this network.

[0067] S3: Train, validate, and test the Transformer-based real-time object detection model TIPS-DETR. Train the TIPS-DETR model using the training set, employing the AdamW optimizer with a cosine annealing learning rate strategy for backpropagation, iterating 300 times. Evaluate the mAP value on the validation set in each iteration, save the optimal number of iterations, and calculate model parameters and running speed. Train mainstream models YOLOv5S, YOLOv5M, YOLOv8S, YOLOv8M, YOLOv10M, and RT-DETR using the same data, and compare the detection accuracy and counting accuracy of the mainstream models and the TIPS-DETR model to evaluate the state-of-the-art performance of the TIPS-DETR model. Based on the RT-DETR model, progressively add improved modules to form Model 1 (RT-DETR), Model 2 (Model 1 + dual-branch fusion backbone), Model 3 (Model 2 + small target enhancement feature pyramid), and Model 4 (Model 3 + optimized loss function, i.e., TIPS-DETR), to verify the impact of each module on the wheat seedling leaf tip detection model. The detection accuracy and counting accuracy of the TIPS-DETR model were tested using a test set and challenging data to evaluate the model's effectiveness. Evaluation metrics included precision, recall, mean accuracy (mAP), number of parameters, GFLOPs (Global Floors), and inference speed. The calculation formulas are as follows:

[0068]

[0069]

[0070]

[0071] Where TP represents a positive sample predicted as positive by the model, TN represents a negative sample predicted as negative by the model, FP represents a negative sample predicted as positive by the model, and FN represents a positive sample predicted as negative by the model. P and R represent Precision and Recall, respectively. mAP 50 This is used to measure the average accuracy of the model across different categories and with an IoU (Intersection over Union) threshold of 0.5. mAP 50-95The average value of the model across different categories and IoU thresholds ranging from 0.5 to 0.95, with a step size of 0.05, is used to measure the model's size. Parameters measure the model's complexity and computational resources required for execution. Speed ​​measures the time the model takes to infer one image. Table 2 shows the experimental results comparing TIPS-DETR with the mainstream YOLO series models. The table clearly shows that TIPS-DETR has a significant advantage in detection accuracy, particularly in accuracy and mAP. 50 and mAP 50-95 The accuracy rates reached 81.23%, 78.76%, and 38.21% respectively, the highest among all compared models. In terms of parameter count and computational cost, TIPS-DETR had the lowest among models of the same scale, and was close to models with even lighter computational loads, achieving a two-way optimization of detection accuracy and computational efficiency. Figure 3 As can be seen, the TIPS-DETR count correlation reached 0.8822, which is superior to other comparative models, further verifying the model's advanced performance in detecting and counting wheat seedling leaf tips. Table 3 shows the model ablation experiment results. As can be seen from the table, after gradually adding the improved modules, the number of model parameters and computational cost gradually decreased, while the model's detection accuracy continuously improved. Ultimately, the improved model reduced the number of parameters by 38.8% and the computational cost by 35.3% compared to the original model, while also improving accuracy and mAP. 50 and mAP 50-95 The improvements were 6.3%, 6.7%, and 5.1%, respectively. Tables 4 to 6 show the test results of the model on different challenging test datasets. The best-performing comparison model, YOLOv8m, the improved original model RT-DETR, and TIPS-DETR were tested on different challenging test datasets. Table 4 shows the model's test results on test set 2, which includes challenging data from complex field environments, including images of straw, weeds, straw + weeds, and images after weeding. Table 5 shows the model's test results on test set 3, which includes challenging data from different lighting and soil conditions, including images of sunny / cloudy days and images with different soil backgrounds. Table 6 shows the model's test results on test set 4, which includes data from different wheat growth stages, including images from the one-leaf to four-leaf stage. As can be seen from the tables, TIPS-DETR maintains high accuracy on various challenging datasets, and from... Figure 4 The visualized detection results show that the model performs excellently on various challenging data. Figure 5The visualization results of different evaluation metrics for TIPS-DETR and mainstream models on test set 4 at different wheat growth stages show that the mainstream comparative models exhibit significant performance degradation at the one-leaf stage (where targets are scarce) and the four-leaf stage (where the background is complex). In contrast, the TIPS-DETR model demonstrates strong robustness, maintaining a stable and higher level of detection accuracy throughout the entire growth period, especially at the one-leaf and four-leaf stages, where its accuracy far surpasses that of the comparative models. Furthermore, the peak values ​​of different metrics in the comparative models show significant divergence across different growth stages; in contrast, TIPS-DETR's evaluation metrics reach their highest global peak at the two-leaf stage, indicating that the model can achieve accurate and comprehensive detection at an earlier two-leaf stage. Figure 6 The visualization results of TIPS-DETR on data collected using smartphones and drones show that TIPS-DETR can effectively detect wheat seedling leaf tips in multi-scale field scenarios on different platforms, demonstrating the model's advanced nature and applicability.

[0072] Table 2 Comparative Experiments

[0073]

[0074] Table 3 Ablation Experiment

[0075]

[0076] Table 4. Data Verification of Difficulties in Complex Field Environments

[0077]

[0078] Table 5. Verification of data on light and soil challenges.

[0079]

[0080] Table 6. Validation of data from different wheat growth stages

[0081]

[0082] S4: The TIPS-DETR real-time target detection model based on Transformer was used to collect visible light images of wheat seedlings using smartphones and drones. After preprocessing according to S1, the images were input into the TIPS-DETR real-time target detection model based on Transformer to obtain the leaf tip detection results of wheat seedlings in the field and count the number of targets.

[0083] The above embodiments are implementation methods of the present invention, but the implementation methods of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. This invention relates to a field wheat seedling leaf tip detection and counting method based on multi-scale feature fusion, characterized in that... Include at least the following steps: S1: Image data acquisition and preprocessing: Visible light images of wheat seedlings were acquired using smartphones and drones. The acquired images were cropped and labeled to construct an image dataset, which was then divided into a training set, a validation set, and a test set. S2: Construct a Transformer-based real-time object detection model, TIPS-DETR, whose framework includes a feature extraction network, a feature fusion network, and a prediction network; ①Feature extraction network: A dual-branch fusion backbone MultiBackbone is adopted. After the input image is mapped and downsampled by the HGStem layer, it enters the heterogeneous feature extraction branch, which leads to the high-resolution feature layer of small targets, denoted as S2. Subsequently The features flow through multiple consecutively stacked spatial-semantic adaptive fusion modules, and finally output four layers of features at different resolutions together with S2, which are denoted as [S2, S3, S4, S5]; ② Feature Fusion Network: Construct a Small Target Enhanced Feature Pyramid (SOEP), and use Spatial Depth Transformation Convolution to fuse the S2 and S3 feature layers; and use the Cross-Stage Local Full-Kernel Module (CSP-OKM) based on cross-stage local networks and full-kernel modules to integrate features, and output a multi-scale predicted feature representation set through bidirectional weighted path aggregation from top to bottom and bottom to top. ③ Prediction Network: A visual transformation decoder and decoupled head structure is adopted. The multi-scale prediction feature representation set is input into the visual transformation decoder, the decoded features are introduced into the decoupled prediction head of the parallel branch, and the category confidence score and target bounding box coordinates are output by combining the Shape-IoU loss function. S3: Training, validation, and testing of the Transformer-based real-time object detection model TIPS-DETR; S4: The TIPS-DETR real-time target detection model based on Transformer was used to collect visible light images of wheat seedlings in the field, preprocess them, and input them into the trained TIPS-DETR model to output leaf tip detection results and count the number of targets.

2. The method according to claim 1, characterized in that: In step S1, the specific methods for image data acquisition and preprocessing are as follows: S11: After wheat sowing, during the wheat seedling emergence to four-leaf stage, use smartphones and drones to collect image data of wheat seedlings; S12: Perform block cropping on the images acquired by the drone. The number of blocks N is determined by the following formula: Where W and H are the image width and height resolution, respectively, and S is the step size; S13: Use the annotation tool to make rectangular annotations. The annotation strategy is to take the top of the tip of the blade as a corner point of the rectangle. The diagonal of the rectangle coincides with or is parallel to the central axis of the blade and extends along the blade direction. The length of the diagonal is 1-2cm. The length of the short side of the rectangle adaptively covers the blade tip outline. S14: After data cleaning of the cropped and labeled images, they are randomly divided into training set, validation set and test set 1. In addition, multiple challenging test sets are manually selected and constructed.

3. The method according to claim 2, characterized in that: The test sets include several challenging sets: Test set 2 contains challenging data from complex field environments; Test set 3 contains challenging data from different light and soil conditions in the field; and Test set 4 contains data from different growth stages of wheat.

4. The method according to claim 1, characterized in that: In step S2, the feature extraction network is a dual-branch fusion backbone, and its specific implementation process is as follows: S211: HGStem layer processing: The input image is downsampled by a convolutional module and then divided into two paths. One path extracts local features through a series of convolutional modules, while the other path retains global features using max pooling. The two feature paths are concatenated and then output through a secondary downsampling module. S212: Heterogeneous feature extraction branch: The output features of HGStem are synchronously fed into the parallel C2f module and HGBlock module to extract high-resolution spatial and semantic information respectively, and the features and high-resolution feature layers of small targets passed to the downstream deep network are denoted as S2. S213: For the features of the S2 layer, a heterogeneous routing strategy is adopted: the direct output of the S2 layer comes only from the fused output of the C2f branch, and the features output by the HGBlock module are only passed to the downstream deep network. S214: SSAFM module processing: It contains two parallel heterogeneous paths. The first path consists of C2f and regular convolution in series, and the second path consists of HGBlock and depthwise separable convolution in alternation. The features of the two paths are aligned and fused at the end of the SSAFM module through the multi-scale adaptive spatial attention gate mechanism MASAG. The MASAG processing flow includes a multi-scale feature fusion stage, a spatial selection stage for calculating weights using Softmax, a spatial interaction stage for interactive modulation using Sigmoid, and a recalibration stage for generating attention maps using pointwise convolution for recalibration. Through the layer-by-layer processing of multiple SSAFM modules, it finally outputs four layers of features with different resolutions together with the high-resolution feature layer of small targets, which are denoted as [S2, S3, S4, S5].

5. The method according to claim 1, characterized in that: In step S2, the feature fusion network is the Small Target Enhanced Feature Pyramid (SOEP), and its specific implementation process is as follows: S221: The S2 feature layer is spliced ​​along the channel axis and compressed to the same scale as the S3 layer after passing through SPDConv. Then it is spliced ​​and fused with the S3 layer features to output feature F3. S222: Flatten the S5 feature layer and output feature F5 through an intra-scale feature interaction module that includes a feedforward neural network and a multi-head attention mechanism; S223: Introduce the CSP-OKM module at feature intersection nodes. S224: After mapping the number of channels of features [F3, S4, F5] to 256 dimensions using several 1x1 convolutions, it enters a bidirectional weighted path containing the RepC3 module and the CSP-OKM module; Top-down path: F5 is upsampled and concatenated with S4, and then sent to the RepC3 module to generate transition feature P4'; P4' is upsampled and concatenated with F3, then fused by the CSP-OKM module and RepC3 module to output high-resolution prediction feature P3; Bottom-up path: P3 is downsampled by SPDConv and concatenated with P4', then fused by RepC3 to generate medium-resolution prediction feature P4; P4 is downsampled by SPDConv and concatenated with F5, then fused by RepC3 module to output low-resolution prediction feature P5; Finally, the network outputs a multi-scale prediction feature representation set [P3, P4, P5], which is then input into the subsequent prediction network.

6. The method according to claim 1, characterized in that: In step S2, the prediction network includes an optimized loss function, Shape-IoU. Feature sets at different spatial scales are input into the visual transformation decoder. Iterative correction is performed using multi-head self-attention, cross-attention, and feedforward mapping networks, combined with two-dimensional position encoding and residual connections. The decoded and refined object query sequence is introduced into a decoupled prediction head, and mapping and parsing are performed through mutually independent parallel classification and regression branches. The bounding box regression branch is calculated based on Shape-IoU, and its mathematical expression is as follows: in, and These represent the predicted bounding box and the ground truth bounding box, respectively. and These represent the center coordinates of the predicted bounding box and the ground truth bounding box, respectively. The length of the diagonal of the smallest bounding closure region that simultaneously contains both the predicted and ground truth bounding boxes; In the item, and These are adaptive weighting coefficients introduced to address changes in the target scale; In the item, It measures the relative deviation between the predicted bounding box and the ground truth bounding box in the width and height dimensions. Used to control the sensitivity of shape penalty.

7. The method according to claim 1, characterized in that: Step 3 specifically includes: training the TIPS-DETR model using the pre-defined training set and saving the optimal model weights; comparing the detection accuracy and counting accuracy of the mainstream model and the TIPS-DETR model to evaluate the advancement of the TIPS-DETR model; conducting ablation experiments to verify the impact of each module; and using the test set and challenging data to test the detection accuracy and counting accuracy of the TIPS-DETR model to evaluate the effectiveness of the model.

8. The method according to claim 1, characterized in that: In step S3, the model is trained using the training set, and backpropagation is performed using the AdamW optimizer in conjunction with a cosine annealing learning rate strategy for 300 iterations. In each iteration, the mAP value is evaluated on the validation set, the optimal number of iterations is saved, and the model parameters and running speed are calculated. The detection accuracy and counting accuracy of the TIPS-DETR model are tested using the test set to evaluate the effectiveness of the model.

Images