Multispectral image semantic segmentation method based on deep learning

Abstract

The invention relates to the field of semantic segmentation, and particularly discloses a multi-spectral image semantic segmentation method based on deep learning, which comprises the following steps: acquiring a target image; selecting red, yellow and blue wave bands from the target image to construct a pseudo-color image; a pre-training super-resolution generative adversarial network based on an RRDBNet architecture is adopted, and a discrimination enhancement operation is executed on the pseudo-color image; performing boundary labeling on the pseudo-color image after the discrimination rate enhancement to obtain a labeled image; and based on an improved UNet model, performing semantic segmentation on the annotated image. In the scheme, on a 16-channel multispectral data set taking betula luminifera seedlings as objects, the method combines a single multispectral camera with image enhancement in deep learning and a semantic segmentation technology, and high-precision segmentation of the plant seedlings can be realized without depending on high-resolution RGB (Red, Green, Blue) imaging equipment.

Description

Technical Field

[0001] This application relates to a semantic segmentation method, specifically a deep learning-based multispectral image semantic segmentation method, belonging to the field of semantic segmentation technology. Background Technology

[0002] *Betula glabra*, a superior broad-leaved tree species endemic to China, has become an important afforestation species in temperate and subarctic regions due to its excellent timber properties and wide application in ecological restoration and afforestation. It has also been introduced and promoted in many countries worldwide. In recent years, the precision breeding and cultivation of *Betula glabra* seedlings has become a research hotspot. Precision breeding not only helps improve the growth rate and stress resistance of seedlings but also enhances the overall quality of seedlings, providing strong support for large-scale forestry production and ecological restoration. High-throughput, high-precision phenotypic trait extraction relies on the automatic segmentation of plant organs from accurate data. The rise of deep learning and image processing technologies has provided new tools for tree species phenotypic recognition and analysis. With the help of efficient image acquisition and analysis systems, dynamic monitoring of the growth status of *Betula glabra* seedlings can be achieved, significantly improving breeding efficiency and the scientific basis of decision-making.

[0003] In the field of plant phenotypic analysis, high-precision semantic segmentation is crucial for plant growth monitoring and trait extraction. Multispectral imaging technology can provide rich spectral information, but due to its inherent low spatial resolution, directly acquired multispectral data often falls short of the requirements for fine segmentation and cannot be effectively applied to plant growth trait detection. Traditional methods typically combine high-resolution RGB imaging equipment to compensate for this deficiency, but this not only significantly increases experimental costs and the complexity of data fusion but also limits the flexibility of field applications to some extent. Summary of the Invention

[0004] To address the problems in the existing technology, this application provides a deep learning-based multispectral image semantic segmentation method.

[0005] The objective of this application can be achieved through the following technical solutions: A deep learning-based multispectral image semantic segmentation method includes: acquiring a target image; A pseudo-color image is constructed from the target image by selecting the red, yellow, and blue bands. A pre-trained super-resolution generative adversarial network based on the RRDBNet architecture is used to perform resolution enhancement operations on the pseudo-color image; Boundary annotations are performed on the resolution-enhanced pseudo-color image to obtain an annotated image; The annotated image is semantically segmented based on the improved UNet model.

[0006] Preferably, the step of using a pre-trained super-resolution generative adversarial network based on the RRDBNet architecture to perform resolution enhancement operations on the pseudo-color image includes: using a pre-trained super-resolution generative adversarial network based on the RRDBNet architecture to perform a 4x resolution enhancement operation on the pseudo-color image; wherein the pre-trained super-resolution generative adversarial network based on the RRDBNet architecture includes multiple residual dense modules, each module having 3 input / output channels, 64 feature channels, and 32 growth channels.

[0007] Preferably, the step of performing boundary annotation on the resolution-enhanced pseudo-color image to obtain an annotated image includes: using LabelMe software to annotate all visible areas of the *Betula platyphylla* seedling tissue in the resolution-enhanced pseudo-color image to ensure complete boundary delineation, and simultaneously generating a corresponding JSON format annotation file; The labeled file is converted into a mask image, and then uniformly scaled to the original image resolution to obtain the labeled image.

[0008] Preferably, the semantic segmentation of the labeled image based on the improved UNet model includes: selecting the UNet model with the visual geometry group as the backbone as the basic framework, introducing a spatial-channel attention module, a hollow spatial pyramid pooling module, a residual module, and a block dropout module to perform semantic segmentation on the labeled image, so as to enhance the model's ability to extract key features and its overall robustness.

[0009] Preferably, the spatial-channel attention module includes two sub-modules: channel attention and spatial attention; The spatial-channel attention module adopts a sequential structure after the channel to ensure that the model completes feature selection in the channel dimension first, and then performs spatial saliency modeling, which helps the model to better capture multi-level information of the target region. The spatial-channel attention module has a compression ratio of 16 and is embedded after each downsampling layer, which enables the model to gradually aggregate salient features during the encoding stage, thereby enhancing the overall segmentation performance and the model's generalization ability.

[0010] Preferably, in the hollow space pyramid pooling module, the dilation rate of the hollow convolution branch is set to 12, 16 and 18 to further expand the receptive field of the model, thereby enhancing the modeling ability of targets at different scales such as crown, leaves and stem. The void space pyramid pooling module effectively extracts multi-level information from 16-channel multispectral images by integrating multiple dilatation rate branches and global average pooling branches. It also improves the model's segmentation accuracy for small structures such as crowns and leaves by feature stitching and 1×1 convolutional layers to fuse multi-scale features.

[0011] Preferably, the convolution kernel size of the residual module is set to 3×3, and the padding is set to 1; The shortcut connection branch of the residual module introduces a 1×1 convolutional layer and a batch normalization layer to automatically align feature channels and ensure dimensionality consistency between the input and output of the residual module.

[0012] Preferably, the core parameters of the block discarding module are set to block_size=3 and drop_prob=0.1 to ensure effective regularization while controlling the range of information loss.

[0013] Preferably, the method further includes: The improved UNet model is trained, and the trained weights are applied to the test set for prediction. The segmentation performance of the model is evaluated by comparing the prediction results with the real mask image, and finally five key evaluation indicators are calculated.

[0014] The beneficial effects of this application are: This application acquires a target image; constructs a pseudo-color image from the target image by selecting red, yellow, and blue bands; performs resolution enhancement on the pseudo-color image using a pre-trained super-resolution generative adversarial network based on the RRDBNet architecture; performs boundary annotation on the resolution-enhanced pseudo-color image to obtain an annotated image; and performs semantic segmentation on the annotated image based on an improved UNet model. In this scheme, on a 16-channel multispectral dataset using *Betula platyphylla* seedlings as the target, this method achieves an average intersection-over-union ratio (mIoU) of 86.75% and an average pixel-level accuracy (mPA) of 93.24%, outperforming traditional image segmentation methods in terms of overall segmentation accuracy, edge preservation ability, and category recognition. Experimental results fully verify the effectiveness and stability of the proposed method in processing low spatial resolution, multi-channel images, demonstrating good generalization ability and practical value. Furthermore, it shows that by combining a single multispectral camera with image enhancement and semantic segmentation techniques from deep learning, high-precision segmentation of plant seedlings can be achieved without relying on high-resolution RGB imaging equipment. Attached Figure Description

[0015] To facilitate understanding by those skilled in the art, the present application will be further described below with reference to the accompanying drawings.

[0016] Figure 1 This is a schematic diagram of the process of a deep learning-based multispectral image semantic segmentation method according to this application; Figure 2 This is a schematic diagram of the seedling label used in this application; Figure 3 This is a schematic diagram of the spatial-channel attention module structure in this application; Figure 4 This is a schematic diagram of the hollow space pyramid pooling module structure of this application; Figure 5 This is a schematic diagram of the block-shaped discard module structure of this application; Figure 6 This is a schematic diagram of the block-shaped discard module structure of this application; Figure 7 The images show the comparison and segmentation results before and after super-resolution reconstruction in this application; Figure 8 This is a comparison chart of the training loss of the model in this application; Figure 9 This is a comparison chart of the segmentation results of the model in this application; Figure 10 This is a comparison chart of the segmentation results of the model in this application; Figure 11 This is a graph showing the specific performance verification results of the model in this application. Detailed Implementation

[0017] The technical solutions of this application will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0018] In order to solve the problems in the existing technology, such as Figure 1 As shown, this application provides a deep learning-based multispectral image semantic segmentation method, including: S101, acquire the target image.

[0019] For example, in order to acquire the target image, images of *Betula platyphylla* seedlings can be captured in this embodiment. To obtain high-quality image data, a uniform and stable artificial light source is set up while natural light interference is shielded. Background and lighting conditions are strictly controlled to minimize the impact of shadows, reflections, and stray light on the image, thereby significantly improving image clarity, contrast, and repeatability. This has become a relatively common and effective operational strategy in current plant image acquisition. Based on this, low-reflectivity black velvet cloth can be used as the background material to further suppress interference caused by background clutter and light source scattering, enhancing image clarity and contrast.

[0020] During the shooting process, the camera lens was aligned with the center of the seedling pot, the shooting height was fixed at approximately 35 cm above the seedling pot, and the horizontal distance between the lens and the sample was 90 cm. Considering that the selected samples were all in the seedling stage, with a plant height of approximately 18-50 cm, this shooting height and distance could comprehensively cover the overall structure of seedlings of different heights, while balancing the field of view and image resolution. This configuration avoided local distortion and background interference caused by close-up shooting, and could clearly present the details of branches and leaves, providing a good foundation for subsequent phenotypic feature extraction. The light source was set directly in front of the sample, using a full-spectrum LED photography light with a color temperature of 6500 K and a power of approximately 50 W, to ensure uniform illumination and minimize the effects of shadows and specular reflections. A total of 350 images were captured during the entire acquisition process. After quality screening (removing blurry, abnormally exposed, or unevenly lit images), 300 valid images were finally retained to construct the original dataset.

[0021] For example, images can be acquired by a real-time multispectral imaging area array camera (16 bands-Vis), with the raw data stored in RAW+HDR format. The RAW file records the unprocessed raw spectral data, while the HDR file contains configuration information describing the RAW data structure. This format can completely preserve all spectral information of the image and is suitable for multispectral imaging analysis. However, the RAW+HDR format is not directly compatible with deep learning frameworks (such as PyTorch or TensorFlow), limiting its direct application in model training and inference. To address this issue, ENVI 5.6 and IDL 8.8 can be used to parse the RAW+HDR data, and IDL batch processing scripts can be written to achieve automatic batch conversion of multispectral data to lossless TIFF format. This conversion process ensures the integrity of image information while improving the usability of data in deep learning models, providing high-quality data support for subsequent semantic segmentation training and evaluation.

[0022] S102, Select the red, yellow and blue bands from the target image to construct a pseudo-color image.

[0023] For example, the target image can be band-selected, with channels 4, 8, and 1 precisely selected, corresponding to the red, green, and blue bands respectively, and then combined into a pseudo-color image. The format of the pseudo-color image can include various formats, such as JPG, PNG, etc.

[0024] S103, a pre-trained super-resolution generative adversarial network based on the RRDBNet architecture is used to perform resolution enhancement operations on the pseudo-color image.

[0025] For example, in one implementation, it may include: A pre-trained super-resolution generative adversarial network based on the RRDBNet architecture was used to perform a 4x resolution enhancement operation on the pseudo-color image. The pre-trained super-resolution generative adversarial network based on the RRDBNet architecture includes multiple residual dense modules. The number of input and output channels of each module is set to 3, the number of feature channels is 64, and the number of growth channels is 32, which effectively improves the resolution of the seedling image and provides clearer visual information for subsequent annotation work.

[0026] S104, perform boundary annotation on the resolution-enhanced pseudo-color image to obtain the annotated image.

[0027] For example, in one implementation, it may include: LabelMe software was used to annotate all visible tissue regions of *Betula platyphylla* seedlings in the enhanced pseudo-color image to ensure complete boundary delineation and to simultaneously generate corresponding JSON format annotation files. The labeled file is converted into a mask image, and then uniformly scaled to the original image resolution to obtain the labeled image.

[0028] All labeled files in this application were converted into mask images and then uniformly scaled to the original image resolution. Finally, the constructed dataset was organized and stored in VOC2007 format. This format provides standardized, high-quality supervision information for the subsequent training and evaluation of semantic segmentation models, laying a solid data foundation for the successful conduct of the research. Seedling label images are shown below. Figure 2 As shown.

[0029] S105, based on the improved UNet model, perform semantic segmentation on the labeled image.

[0030] For example, the semantic segmentation of the labeled image based on the improved UNet model includes: The UNet model, with the Visual Geometry Group (VGG) as its backbone, is selected as the basic framework. Spatial-channel attention module, hollow spatial pyramid pooling module, residual module and block dropout module are introduced to perform semantic segmentation on the labeled image to enhance the model's ability to extract key features and its overall robustness.

[0031] UNet is a convolutional neural network widely used in image segmentation tasks. Its symmetrical encoder-decoder structure, combined with skip connections, enables UNet to efficiently integrate multi-layer feature information while maintaining spatial resolution, making it particularly suitable for fine segmentation tasks with high requirements for edges and details.

[0032] Considering that the data described in this application consists of multi-channel, low-resolution multispectral images, and that the original UNet model is mainly designed for 3-channel or single-channel JPG / PNG images, it is difficult to directly adapt to the multispectral data collected in this application. Therefore, this application makes structural improvements on the UNet model with VGG as its backbone to achieve effective input and deep feature extraction of multispectral images. Specifically, the input layer of the model is first expanded, upgrading the original structure that only supports single-channel or RGB three-channel input to accept 16-channel multispectral image input. To this end, the channel mapping relationship is reconstructed, and the data loading and preprocessing process is rewritten to ensure that the model can be compatible with multispectral high-dimensional data parsed in TIFF format. In the encoder part, in order to improve the model's ability to perceive and express multi-channel features, the structure and parameter configuration of each level of convolution module are redesigned, including the convolution kernel size adapted to the number of input channels, the normalization strategy of the activation function, and the dimensional connection and compression path of the feature map between different levels. The number of parameters of the model changes after the adjustment (Table 1). The aforementioned adjustments enable the model to fully extract and retain high-dimensional semantic features across channels in multispectral images while maintaining the integrity of the spatial structure, providing a solid feature foundation for achieving high-precision semantic segmentation. Furthermore, this paper introduces four enhancement modules into the improved model: spatial-channel attention mechanism, hollow spatial pyramid pooling, residual module, and block dropout, to further enhance the model's ability to extract key features and its overall robustness.

[0033] Table 1 For example, to further improve the segmentation performance of the model in complex scenarios involving *Betula spp.* seedlings, this application introduces a Convolutional Block Attention Module (CBAM). The CBAM module integrates two sub-modules: channel attention and spatial attention. It aims to guide the model to adaptively focus on more discriminative feature regions in both channel and spatial dimensions, thereby improving segmentation accuracy (e.g., ...). Figure 3 (As shown).

[0034] For the segmentation task of multispectral images of *Betula platyphylla* seedlings, characterized by high background noise, rich plant structural details, and significant differences in the contribution of different channels to the target, the channel attention module dynamically assigns weights to input channels, enabling the model to automatically select channels more sensitive to leaf and stem segmentation while suppressing noisy channels that contribute less to the segmentation task, thus effectively reducing background interference. The spatial attention module further guides the model to focus on the small target region of the seedling in the spatial dimension, effectively mitigating interference from complex backgrounds and shadows, and significantly improving the segmentation accuracy of the canopy and leaf edges. The CBAM module adopts a serial structure (channel first, then spatial), ensuring that the model completes feature selection in the channel dimension first, and then performs spatial saliency modeling, which conforms to the human visual perception process and helps the model better capture multi-level information of the target region. In this embodiment, the compression ratio of CBAM is set to 16, and the entire module is embedded after each downsampling layer, allowing the model to gradually aggregate salient features during the encoding stage, enhancing the overall segmentation performance and the model's generalization ability.

[0035] For example, such as Figure 4 As shown, Atrous Spatial Pyramid Pooling (ASPP) is an effective dilated convolutional structure. By setting dilated convolutional branches with different dilation rates, it extracts multi-scale features in parallel, effectively enhancing the model's ability to perceive targets at different scales.

[0036] For the segmentation task of multispectral images of *Betula platyphylla* seedlings, the ASPP module can fully exploit the multi-scale feature representation capabilities of multispectral images in both spatial and spectral dimensions. Multispectral images contain rich spectral information, and different channels exhibit heterogeneous features in the boundaries and textures of plant organs such as crowns, leaves, and stems. Traditional convolution operations often rely on a fixed receptive field, making it difficult to simultaneously consider the fine-grained structure of seedlings and global background information, resulting in limited segmentation performance in complex scenes. To better capture the multi-scale features of plant seedlings in multispectral images, this embodiment sets the dilation rate of the dilated convolution branch in the ASPP module to 12, 16, and 18 to further expand the model's receptive field, thereby enhancing its ability to model targets at different scales such as crowns, leaves, and stems. By setting a larger dilation rate, the model can fully integrate local details and global contextual information while ensuring controllable computational complexity, thus significantly improving the target segmentation performance in complex backgrounds. In addition, the ASPP module effectively extracts multi-level information from 16-channel multispectral images by integrating multiple dilatation rate branches and global average pooling branches. It also achieves multi-scale feature fusion through feature stitching and 1×1 convolutional layers, further improving the model's segmentation accuracy for small structures such as crowns and leaves, and significantly enhancing the model's segmentation performance and adaptability to complex scenes.

[0037] For example, the residual module is a core component in the deep residual network ResNet, which can solve the problems of vanishing and exploding gradients in deep neural networks. It achieves cross-layer feature transfer by introducing "skip connections" to directly pass input features to subsequent layers.

[0038] For the segmentation task of multispectral images of *Betula platyphylla* seedlings, the residual module can fully exploit the deep feature representation capabilities of multispectral images while ensuring the trainability and stability of the model. The 16 channels of a multispectral image exhibit significant heterogeneity in the boundary and texture representation of plant organs such as crown, leaves, and stems. Deep convolutional networks are prone to gradient vanishing and model degradation when extracting these complex features, affecting the accuracy of segmentation and the restoration of fine-grained structures. This embodiment sets the convolutional kernel size to 3×3 and the padding to 1 to effectively extract local features while maintaining the feature map spatial resolution. Furthermore, to address the issue of inconsistent input and output channel numbers in multispectral image segmentation tasks, this embodiment introduces a 1×1 convolutional layer and a batch normalization (BN) layer in the shortcut connection branch of the residual module to automatically align feature channels. This ensures dimensionality consistency between the input and output of the residual module, avoids dimensionality conflicts during feature fusion, and improves the model's convergence speed and stability (e.g., ...). Figure 5 ).

[0039] For example, the DropBlock module is an improved regularization method specifically designed for processing feature maps in convolutional neural networks. Unlike the traditional Dropout method, which randomly discards individual neurons, DropBlock effectively suppresses the convolutional layers' dependence on local regions by randomly removing contiguous regions (i.e., "blocks") from the feature map.

[0040] For the semantic segmentation task of multispectral images of *Betula platyphylla* seedlings, the data channels are numerous and the spectral information is complex. Furthermore, the seedlings exhibit rich and detailed morphological features in their spatial structure. Simultaneously, significant noise exists in the background region, and its uneven distribution further increases the difficulty of the segmentation task. In such complex scenarios, traditional regularization methods struggle to fully model spatial structural features, easily leading to overfitting during training or neglecting crucial contextual information. To improve the model's generalization and feature learning capabilities in complex backgrounds, this embodiment employs the DropBlock regularization module. This module randomly discards continuous spatial regions in the feature map during the training phase, guiding the model to learn more discriminative global and local features even with missing local information. This significantly enhances the model's ability to perceive multi-scale targets (such as canopy, leaves, and stems). In this embodiment, the core parameters of the DropBlock module are set to block_size=3 and drop_prob=0.1 to ensure effective regularization while controlling the range of information loss. Furthermore, DropBlock's structured dropout strategy requires no additional parameters or computational overhead, achieving regularization solely through spatial block-level occlusion of the input feature map. This makes it particularly suitable for resource-constrained multispectral image segmentation tasks with high real-time inference requirements. This mechanism helps the model extract effective features from richer context, improving robustness and segmentation accuracy for fine-grained targets while maintaining lightweight design. In this embodiment, the DropBlock module serves as a key technology for improving the semantic segmentation performance of multispectral images of *Betula platyphylla* seedlings. Its structural diagram is shown below. Figure 6 As shown.

[0041] Furthermore, in another implementation of this embodiment, the method further includes: The improved UNet model is trained, and the trained weights are applied to the test set for prediction. The segmentation performance of the model is evaluated by comparing the prediction results with the real mask image, and finally five key evaluation indicators are calculated.

[0042] For example, in semantic segmentation tasks, the original UNet model is typically trained using the multi-class cross-entropy loss function. This loss function effectively improves pixel-level classification accuracy by maximizing the consistency between the predicted probability and the true label on a pixel-by-pixel basis. However, the optimization objective of cross-entropy deviates somewhat from the IoU (Intersection over Union) commonly used to evaluate semantic segmentation performance, especially in scenarios with severely imbalanced class distributions or a large number of small targets. This inconsistency may cause the model to overlook semantically important classes that have a small pixel footprint.

[0043] In this application, to address the imbalance in the proportion of branches, leaves, and background in images of *Birchia glabra* seedlings, a Lovász-Softmax loss function is introduced to more directly optimize the mIoU (Mean IoU) metric. A comparison with the cross-entropy loss function is shown in Table 2. This loss function is constructed based on the Lovász extension theory and is a differentiable approximate upper bound of IoU. Therefore, mIoU can be used as a direct optimization target during training, achieving end-to-end optimization from input to output. Compared to the pixel-level independence assumption of traditional cross-entropy, Lovász-Softmax calculates the ranking error at the whole-image level, making it particularly suitable for handling imbalanced classes and sparse targets. It exhibits higher robustness and accuracy for small targets (such as branch edges) in the semantic segmentation of *Birchia glabra* seedlings.

[0044] Furthermore, the main parameter settings of the Lovász-Softmax loss function used in this study are as follows: 1. per_image=True: Calculate the loss independently for each image and then take the average, which helps to maintain stronger constraints on the details of each seedling image during training; 2. classes='present': Only calculate the loss for classes that actually exist in the current image, avoiding the introduction of noise for classes that do not appear; 3. ignore=0 (or other background index): can flexibly skip background labels, allowing the model to focus on the plant structure itself and improve foreground segmentation performance.

[0045] These settings make Lovász-Softmax not only theoretically more aligned with the mIoU evaluation objective, but also highly practical and scalable. Specifically, the Lovász-Softmax loss is defined as follows: Where: C is the set of categories. and They are categories The predicted and actual values.

[0046] Table 2 Comparison of Loss Functions The experimental comparison results of this application will be introduced below with reference to specific experiments.

[0047] Three hundred samples of *Betula spp.* seedlings with different postures were selected as the data basis for model training and performance evaluation. To ensure the model's generalization ability during training and to objectively evaluate its performance, the dataset was divided into a training set and a test set in an 8:2 ratio, following standard practice. 240 samples were used for model training, and 60 samples were used for final testing to evaluate the model's actual generalization ability on unseen samples. This data partitioning strategy effectively improved the scientific rigor of the model evaluation and the reliability of the experimental results, providing stable performance assurance for subsequent promotion in practical applications.

[0048] Six evaluation metrics were used to measure the segmentation performance of the model: mPA (mean pixel accuracy), mIoU (mean intersection-over-union ratio), Recall, Precision, F1 Score, and Accuracy. These metrics evaluate the accuracy and robustness of the segmentation results from different perspectives, ensuring the model's ability to identify seedling tissue regions.

[0049] mPA measures the model's classification accuracy across all classes, then averages the results. The formula is as follows: Where N is the total number of categories, M TP M represents the number of true positive pixels, i.e., the number of pixels that are actually foreground and predicted to be foreground. FP M represents the number of false positive pixels, i.e., the number of pixels that are actually background but are predicted as foreground. TN M represents the number of true negative pixels, i.e., the number of pixels that are actually Beijing but are predicted to be the background. FN This represents the number of false negative pixels, i.e., the number of pixels that are actually foreground but are predicted as background.

[0050] mIoU measures the degree of overlap between the predicted and ground truth regions, and it is the most commonly used metric in segmentation tasks. The calculation formula is as follows: Recall measures a model's ability to correctly identify all true foreground objects, and is calculated using the following formula: Precision measures how many pixels predicted as foreground by the model are actually foreground elements (branches and leaves). It is defined as follows: The F1 score is the harmonic mean of precision and recall, and it comprehensively evaluates the segmentation ability of a model. Its definition is as follows: Accuracy measures the proportion of samples correctly predicted by a model out of the total sample. It is defined as follows: Here, Number of Correct Predictions represents the number of correctly predicted pixels, and Total Number of Predictions represents the total number of pixels.

[0051] To deeply analyze the effect of the super-resolution generative adversarial network Real-ESRGAN in improving the segmentation of *Birchia pulcherrima* seedlings, a baseline UNet model was used as the basis. The original pseudo-color image and its super-resolution processed image were manually annotated to generate corresponding mask images as the sole input variables. The impact of mask images of different resolutions on semantic segmentation accuracy was compared and analyzed. After super-resolution processing, the model's performance was further improved, with mIoU increasing by 4.33% and mPA increasing by 3.40%, indicating that higher-quality input images help enhance segmentation accuracy and class discrimination ability. The comparison and segmentation results before and after super-resolution reconstruction are shown below. Figure 7 As shown in Table 3.

[0052] Table 3. Segmentation results of the model before and after super-resolution reconstruction. To systematically evaluate the effectiveness of each module in the improved UNet model, based on the VGG16 backbone UNet model, the CBAM attention mechanism (C), residual structure (R), ASPP hollow spatial pyramid pooling module (A), and DropBlock regularization module (D) were gradually introduced. Multiple ablation experiments covering single-module, dual-module, triple-module, and quadruple-module combinations were designed. Experiments were conducted under a unified dataset and parameter configuration, recording the mIoU, mPA, FPS, and parameter size (Params) for each model version to comprehensively measure the impact of each module on performance, speed, and complexity. The effectiveness analysis of each module is shown in Table 4.

[0053] First, introducing each module individually can improve model performance to some extent. Specifically: the CBAM module (UNet-C) significantly improves pixel-level classification accuracy, achieving an mPA of 81.371, especially in the more refined and accurate classification at stem-leaf junctions and edge regions. The ASPP module (UNet-A) enhances the model's ability to perceive different seedling structures through multi-scale dilated convolutions, increasing mIoU from 77.193 to 79.632. Furthermore, the DropBlock module, as a lightweight regularization method, improves the model's generalization ability without parameter growth, achieving an mIoU of 79.354 and an mPA of 81.026, making it suitable for distributed phenotypic monitoring platforms with high real-time requirements. In contrast, while the residual structure introduces some parameter overhead (increasing the number of parameters to 27.982M), it effectively alleviates the feature degradation problem of deep networks in complex backgrounds, improving the overall stability of the network.

[0054] In the module combination experiments, different structures exhibited good complementary characteristics, significantly enhancing the model's ability to identify the fine structure of *Betula sclerophylla* seedlings. Taking UNet-RD (Residual + DropBlock) as an example, it outperformed any single-module model in both mIoU (81.795) and mPA (83.819), while achieving a good balance between maintaining a moderate number of parameters (27.982M) and a relatively high speed (58.462 FPS). Among the three-module combinations, the UNet-RAD model performed particularly well in handling complex situations such as small leaves, overlapping structures, and weak boundaries, achieving an mPA of 90.368, the highest among all three-module combinations, indicating the strongest expressive power under the synergy of feature enhancement, scale modeling, and regularization suppression mechanisms; while UNet-CRA was the best in mIoU (83.375), capturing the contour of the overall structure more completely. The final UNet-CRAD model, integrating all four modules, achieved optimal performance in semantic segmentation accuracy for *Betula sylvestris* seedlings, with mIoU increasing to 86.751 and mPA reaching 93.240, representing improvements of 9.56% and 25.44% respectively compared to the base UNet. Although the introduction of modules increased resource overhead, with the number of parameters increasing from 21.783 to 36.938M and inference speed decreasing from 59.746 FPS to 44.650 FPS (a decrease of approximately 25.28%), the model showed significant improvements in edge sharpness, class separation, and robustness, making it particularly suitable for low-resolution multispectral fine segmentation tasks.

[0055] Table 4 Module Validity Analysis To comprehensively evaluate the impact of multi-channel input on the model's segmentation performance, comparative experiments with different numbers of input channels were designed in the experimental section, systematically exploring the relationship between the number of input channels and segmentation accuracy (Table 5). With the increase in the number of input channels, the model's segmentation performance showed a significant upward trend. Specifically, when the number of input channels increased from 3 to 8, the model's mIoU improved from 82.71% to 84.28%, a relative improvement of approximately 1.90%; mPA improved from 84.82% to 90.47%, a relative improvement of approximately 6.66%. When the number of input channels further increased to 16, the model's mIoU and mPA further improved to 86.75% and 93.24%, respectively. Compared to 3 channels, the cumulative improvement in mIoU was approximately 4.88%, and the cumulative improvement in mPA was approximately 9.94%.

[0056] The results show that increasing the number of input channels effectively enhances the model's ability to perceive multidimensional spectral or multimodal information, helping to capture richer plant structural and textural features, thereby significantly improving segmentation performance. However, an excessive number of channels may also increase model complexity and computational resources, requiring a trade-off based on the specific task and computing environment in practical applications. In conclusion, appropriately increasing the number of input channels has a significant promoting effect on improving plant segmentation performance.

[0057] Table 5. Relationship between the number of channels and module performance. Under the same experimental settings, the performance of five mainstream semantic segmentation models was compared, including PSPNet (ResNet50), HRNet, DeepLabV3+ (MobileNet), UNet, UNet++, and the improved model proposed in this application. All models were trained on a selected dataset of 300 images. The training process consisted of two phases: the first 50 epochs were a freeze phase, where the backbone network remained frozen and only the feature extraction layer was fine-tuned; after 50 epochs, the backbone network was unfrozen for end-to-end training. The loss function decreased rapidly in the first 20 epochs of training, then plateaued, and eventually converged. Figure 8 ).

[0058] To verify the effectiveness and superiority of the proposed improved UNet model in plant phenotypic segmentation, this embodiment selected the aforementioned model as the baseline. Furthermore, the baseline model used images whose resolution was enhanced through a super-resolution model, and trained on the original low-resolution images. Under the same data volume and training strategy, multiple evaluation metrics for each model were analyzed, including mIoU, mPA, Accuracy, and fine-grained metrics such as Recall, Precision, and F1-score for the two categories of stem and leaf. The segmentation results of each model are compared in Table 6. Figure 9 As shown.

[0059] In terms of mIoU, the improved UNet (16-channel input) model proposed in this application performs best, achieving an mIoU of 86.75%, significantly outperforming the traditional UNet (77.19%) and UNet++ (83.05%), while also showing a significant improvement over HRNet (79.21%) and DeepLabV3+ (76.93%). This indicates that by introducing multi-channel hyperspectral information and a structure enhancement module, the model's ability to segment semantic regions is effectively improved. Regarding accuracy, the improved model also performs excellently, reaching 99.71%, a slight improvement over U-Net++ (99.63%) and HRNet (99.52%), demonstrating stronger robustness in pixel-level classification tasks. In terms of fine-grained category segmentation, the improved model achieved the best results in both stem and leaf segmentation: for the stem category, the F1-score reached 86.52%, and the recall and precision were 85.39% and 87.68%, respectively, significantly outperforming other comparative models, especially in terms of recall, which significantly led PSPNet (44.48%) and DeepLabV3+ (76.33%); for the leaf category, the F1-score reached 94.68%, and the precision and recall were 94.88% and 94.49%, respectively, demonstrating the model's excellent perception and segmentation ability for complex structural details.

[0060] From the final segmentation results, the PSPNE model's segmentation performance is relatively weak, only able to roughly outline the overall contour of the seedling, while almost unable to represent the detailed segmentation of leaves and branches. In contrast, the Deeplabv3+ model performs well in terms of the continuity of branches and leaves, but lacks detailed depiction of leaf contours, especially in the leaf tip area where clear boundaries are lacking. The HRNet model has ideal overall segmentation results, but exhibits leaf adhesion issues. The UNet model suffers from background misidentification, resulting in noise in the background area; while the UNet++ model performs well overall, but still has problems such as incomplete leaves and horns. Compared to the above models, the improved model proposed in this application not only avoids background missegmentation but also better distinguishes gaps between leaves, resulting in overall segmentation performance superior to other comparative models. The segmentation results of the models are compared below. Figure 10 As shown.

[0061] Table 6 Comparison of Model Segmentation Results At present, there are relatively few studies on plant semantic segmentation based on low-resolution multispectral images. The mainstream plant phenotypic detection still mainly relies on high-resolution RGB images, such as AC-UNet, which uses a VGG16 backbone network to optimize the traditional UNet structure and combines ASPP module and cross attention mechanism (CCA)

[18] to achieve 87.50% mIoU and 93.36% mPA in the segmentation of stems and leaves of birch bark. Another model based on improved UNet and Pix2PixHD also introduces VGG16 backbone and spatial channel attention mechanism (CBAM) and achieves 87.95% mIoU and 93.16% mPA with the support of large-scale training data. Model comparisons show that, compared to the AC-UNet model (4032×3024 / 490 images) and the improved UNet model (4032×3024 / 3600 images) based on high-resolution RGB images, the multi-channel low-resolution multispectral model (512×256 / 300 images) proposed in this application significantly reduces data requirements while maintaining competitive segmentation performance: the total number of pixels is reduced by 97.9% compared to AC-UNet and by 98.9% compared to the improved UNet; the number of samples is reduced by 38.8% compared to AC-UNet and by 91.7% compared to the improved UNet. Comparisons with similar studies are shown in Table 7.

[0062] Table 7 Comparison of similar studies In summary, the improved UNet model proposed in this paper not only outperforms existing mainstream segmentation models in overall performance, but also demonstrates extremely high accuracy and robustness in fine-grained segmentation of plant structures, which can meet the needs of fine extraction of complex structures in high-throughput plant phenotypic analysis.

[0063] To verify the performance of the proposed model in the target data scenario, this study conducted model adaptability tests based on 16-channel multispectral data. The proposed improved UNet model is specifically designed and trained for 16-channel multispectral input data. Its structure and parameter configuration fully consider the feature distribution and information redundancy of this type of data, aiming to better capture and utilize multispectral information. To further evaluate the model's performance, this paper also introduces traditional 3-channel input images and designs cross-modal input tests, systematically comparing and analyzing the model's performance under multispectral input with that under 3-channel input.

[0064] Experimental results show that although the model can effectively handle 3-channel image input without significant structural modifications and outperforms the basic UNet network in overall performance, its performance under 3-channel input conditions is still significantly lower than that under 16-channel multispectral input conditions (Table 8 and...). Figure 11 This result clearly reveals the high structural adaptability of the proposed model to multispectral data and its ability to fully utilize high-dimensional spectral information, verifying the customized advantage of the model being "designed specifically for multichannel data".

[0065] Table 8. Specific performance verification of the model To delve into the intrinsic mechanisms by which multispectral information influences the performance of semantic segmentation models, particularly its specific contributions to segmentation accuracy at the levels of channel selection and band response, this application conducts experimental analysis focusing on the coupling relationship between the spectral characteristics of different bands and plant tissues. The multispectral camera used acquired data across 16 bands, covering the visible light region (478.72 nm to 634.80 nm), which can be divided into 3 blue bands (478.72–489.31 nm), 5 green bands (502.48–564.56 nm), and 8 red bands (601.98–634.80 nm). Considering that the effectiveness of plant image segmentation is not only affected by its color features but also closely related to its intrinsic physiological characteristics, and that these differential features often exhibit significantly different spectral responses across different bands.

[0066] To clarify the role of color features in semantic segmentation of *Betula glabra* seedlings, this application conducted a comparative experiment based on band removal and analyzed it in conjunction with their typical color characteristics. The branches of *Betula glabra* seedlings are typically gray, while the leaves are typically green; this significant color difference mainly stems from the difference in chlorophyll content between tissues. Given this significant difference, the experiment intentionally and gradually removed the 502.48nm, 514.02nm, and 527.59nm bands corresponding to green light, constructing a new dataset. Comparative training was then conducted while maintaining the general structure of the model and the training parameters. The experimental results are shown in Table 9. With the gradual removal of green light bands, the model's segmentation performance in the leaf region significantly decreased, while the impact on the segmentation effect in the branch region was minimal, indicating that the green light band plays a crucial role in the model's differentiation of branch and leaf structures. This finding is highly consistent with existing research conclusions: chlorophyll is mainly distributed in photosynthetic organs such as leaves, and its content in non-photosynthetic tissues such as branches is extremely low. Because chlorophyll has selective spectral absorption characteristics, it mainly absorbs red and blue light bands, while its absorption of green light is relatively weak. Therefore, a large amount of green light is reflected and transmitted, giving the leaves their typical green appearance.

[0067] Building upon this foundation, this application further explored the potential impact of plant physiological characteristics on segmentation performance. The experimental design excluded the green light band (especially around 550 nm), a band highly sensitive to plant physiological states, constructing a new dataset and comparing it with the complete band dataset. The experimental results are shown in Table 9. Under the same model configuration, removing this sensitive band significantly reduced leaf segmentation performance, with both mIoU and mPA decreasing to varying degrees, particularly noticeable in the leaf region. This indicates that the green light band not only expresses color differences but also contains important physiological information reflecting plant nutritional status and metabolic levels. Based on existing research, this information may be related to key physiological indicators such as leaf nutrient accumulation levels, photosynthetic efficiency, and chlorophyll concentration. In particular, leaves, as the most metabolically active organs, tend to accumulate nutrients that support photosynthesis and energy metabolism, while the more structurally robust and metabolically slower branch tissues are less dependent on these physiological indicators. Further analysis revealed that the distribution of phosphorus among these nutrients varied significantly across plant organs, which may be one of the important reasons for the changes in segmentation performance.

[0068] In summary, through band elimination experiments and spectral response analysis, this application systematically reveals the key role of bands in distinguishing the branch and leaf structure of *Betula spp.* seedlings. It not only demonstrates their significant expressive ability in color differences, but also reflects the plant physiological characteristics they carry, verifying the profound impact mechanism of spectral band selection on the performance of semantic segmentation models.

[0069] Table 9 Comparison of the intrinsic mechanisms of segmentation models This application addresses the problems of low spatial resolution in multispectral images, the strong dependence of traditional segmentation methods on high-resolution images, and the complexity of data processing. It proposes a deep learning-based semantic segmentation method for multispectral images. On a 16-channel multispectral dataset using *Betula platyphylla* seedlings as the subject, this method achieves an average intersection-over-union (mIoU) of 86.75% and an average pixel-level accuracy (mPA) of 93.24%, outperforming traditional image segmentation methods in overall segmentation accuracy, edge preservation, and category recognition. Experimental results fully validate the effectiveness and stability of the proposed method in processing low spatial resolution, multi-channel images, demonstrating good generalization ability and practical value.

[0070] This further demonstrates that by combining a single multispectral camera with image enhancement and semantic segmentation techniques from deep learning, high-precision segmentation of plant seedlings can be achieved without relying on high-resolution RGB imaging equipment. This strategy not only effectively reduces the experimental hardware requirements and system complexity but also provides a new approach and feasible path for the on-site, intelligent, and lightweight application of multispectral imaging technology in plant phenotyping research in agriculture, forestry, and other fields. In the future, this method is expected to be widely promoted and applied in various phenotyping analysis scenarios such as crop growth monitoring, pest and disease identification, and variety selection.

[0071] The preferred embodiments disclosed above are merely illustrative of this application. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of this application, thereby enabling those skilled in the art to better understand and utilize this application. This application is limited only by the claims and their full scope and equivalents.

Claims

1. A deep learning-based multi-spectral image semantic segmentation method, characterized in that, include: Acquire the target image; A pseudo-color image is constructed from the target image by selecting the red, yellow, and blue bands. A pre-trained super-resolution generative adversarial network based on the RRDBNet architecture is used to perform resolution enhancement operations on the pseudo-color image; Boundary annotations are performed on the resolution-enhanced pseudo-color image to obtain an annotated image; The annotated image is semantically segmented based on the improved UNet model.

2. The deep learning-based multispectral image semantic segmentation method according to claim 1, characterized in that, The pre-trained super-resolution generative adversarial network based on the RRDBNet architecture performs resolution enhancement operations on the pseudo-color image, including: A pre-trained super-resolution generative adversarial network based on the RRDBNet architecture is used to perform a 4x resolution enhancement operation on the pseudo-color image. The pre-trained super-resolution generative adversarial network based on the RRDBNet architecture includes multiple residual dense modules, each of which has 3 input and output channels, 64 feature channels, and 32 growth channels. According to claim 1, a deep learning-based multispectral image semantic segmentation method is characterized in that the step of performing boundary annotation on the resolution-enhanced pseudo-color image to obtain an annotated image includes: LabelMe software was used to annotate all visible tissue regions of *Betula platyphylla* seedlings in the enhanced pseudo-color image to ensure complete boundary delineation and to simultaneously generate corresponding JSON format annotation files. The labeled file is converted into a mask image, and then uniformly scaled to the original image resolution to obtain the labeled image.

3. The deep learning-based multispectral image semantic segmentation method according to claim 1, characterized in that, The semantic segmentation of the labeled image based on the improved UNet model includes: The UNet model, with the visual geometry group as its backbone, is selected as the basic framework. A spatial-channel attention module, a hollow spatial pyramid pooling module, a residual module, and a block dropout module are introduced to perform semantic segmentation on the labeled images, so as to enhance the model's ability to extract key features and its overall robustness.

4. The deep learning-based multispectral image semantic segmentation method according to claim 4, characterized in that, The spatial-channel attention module includes two sub-modules: channel attention and spatial attention. The spatial-channel attention module adopts a sequential structure after the channel to ensure that the model completes feature selection in the channel dimension first, and then performs spatial saliency modeling, which helps the model to better capture multi-level information of the target region. The spatial-channel attention module has a compression ratio of 16 and is embedded after each downsampling layer, which enables the model to gradually aggregate salient features during the encoding stage, thereby enhancing the overall segmentation performance and the model's generalization ability.

5. The deep learning-based multispectral image semantic segmentation method according to claim 4, characterized in that, The dilation rates of the dilated convolution branches in the dilated spatial pyramid pooling module are set to 12, 16, and 18 to further expand the receptive field of the model, thereby enhancing the modeling ability of targets at different scales, such as crown, leaves, and stems. The void space pyramid pooling module effectively extracts multi-level information from 16-channel multispectral images by integrating multiple dilatation rate branches and global average pooling branches. It also improves the model's segmentation accuracy for small structures such as crowns and leaves by feature stitching and 1×1 convolutional layers to fuse multi-scale features.

6. The deep learning-based multispectral image semantic segmentation method according to claim 4, characterized in that, The kernel size of the residual module is set to 3×3, and the padding is set to 1. The shortcut connection branch of the residual module introduces a 1×1 convolutional layer and a batch normalization layer to automatically align feature channels and ensure dimensionality consistency between the input and output of the residual module.

7. The deep learning-based multispectral image semantic segmentation method according to claim 4, characterized in that, The core parameters of the block-based discarding module are set to block_size=3 and drop_prob=0.1 to ensure effective regularization while controlling the range of information loss.

8. The deep learning-based multispectral image semantic segmentation method according to claim 1, characterized in that, The method further includes: The improved UNet model is trained, and the trained weights are applied to the test set for prediction. The segmentation performance of the model is evaluated by comparing the prediction results with the real mask image, and finally five key evaluation indicators are calculated.

Images