White peony root variety identification method and system based on multi-modal organ feature fusion

Abstract

This invention discloses a method and system for identifying white peony varieties based on multimodal organ feature fusion, comprising the following steps: S1, acquiring and preprocessing root modal images and leaf-flower modal images of white peony; S2, inputting them into a deep learning backbone network containing an organ-perception feature enhancement module to generate organ-enhanced feature maps; S3, constructing independent classification sub-networks based on the organ-enhanced feature maps to predict the variety of root and leaf-flower modal images, generating corresponding classification probability vectors; S4, calculating the prediction uncertainty based on the classification probability vectors and generating fusion weights; S5, weighted fusion of the classification probability vectors of the root and leaf-flower modal images based on the fusion weights to generate a fused prediction probability vector, thereby identifying the white peony variety. This scheme improves the accuracy and stability of white peony variety identification by introducing an organ-perception feature enhancement module and a multimodal dynamic fusion module driven by prediction uncertainty.

Description

Technical Field

[0001] This invention relates to the fields of plant phenotypic recognition, computer vision, and deep learning image recognition technology, and particularly to a method and system for identifying white peony varieties based on multimodal organ feature fusion. Background Technology

[0002] White peony root is a commonly used bulk Chinese medicinal herb, and its quality is significantly affected by factors such as variety, ecological environment of origin, cultivation method, and processing stage. Traditional identification of varieties or sources mainly relies on chemical component detection, microscopic structure observation, or molecular marker analysis. Although the above methods have a certain degree of accuracy, they generally have drawbacks such as complex operation, long detection cycle, destructive nature, and unsuitability for rapid screening of large batches of samples, making it difficult to meet the industry's demand for real-time, non-destructive, and automated quality identification of white peony root.

[0003] With the development of deep learning and computer vision, image-based plant feature recognition technology has been gradually applied to scenarios such as crop variety identification, fruit and vegetable grading, disease identification, and appearance quality assessment of Chinese medicinal materials. However, when performing deep learning recognition on Chinese medicinal materials, existing technologies generally suffer from the following common problems, which make it difficult for existing methods to simultaneously achieve recognition accuracy, stability, and feasibility for large-scale application in tasks such as identifying Chinese medicinal materials like white peony:

[0004] (1) Insufficient information from single-modal images. Different organs of white peony, such as the root topology, leaf vein texture, petal color and shape, all carry information about the differences in varieties. Recognition methods that rely solely on a single image of the root or leaf cannot fully reflect the characteristics of the variety, resulting in insufficient recognition stability. Especially when the number of samples is limited or the shooting conditions are varied, single-modal models often overfit to the features of a certain organ and are difficult to generalize to information of other organs.

[0005] (2) Lack of cross-organ feature representation modeling. Conventional convolutional neural networks only perform uniform convolution extraction on the input image and cannot adaptively enhance based on the structural differences, color features and texture patterns of different organs. This results in key features being easily weakened or omitted, making it difficult to perform targeted enhancement and modeling of organ features such as root topology, leaf texture and petal morphology.

[0006] (3) Single multimodal fusion method. Some studies use simple stitching, fixed weighting or average fusion, but in actual application, there are significant differences in the acquisition conditions of different organ images (such as lighting, background, shooting angle), which leads to low reliability of the prediction results of a certain modality for some samples. Fixed fusion methods are difficult to guarantee the stability of model output in complex scenes. In some samples, the low reliability modality may even dominate the fusion result, thus amplifying the negative impact of noise modality.

[0007] (4) Image quality differences lead to unstable feature extraction. The images of white peony roots, leaves and flowers are often affected by soil residue, uneven lighting and different shooting equipment during the acquisition. Existing recognition methods lack a unified image preprocessing mechanism, which is not conducive to the model obtaining consistent effective features. This also makes it difficult for subsequent feature enhancement and modality fusion modules to work stably on a unified input distribution.

[0008] Therefore, there is an urgent need for a deep learning recognition method that can comprehensively utilize image information from different organs of peony, possess cross-modal adaptive enhancement capabilities, and achieve dynamic fusion based on prediction uncertainty, in order to improve the accuracy and robustness of peony variety identification. Summary of the Invention

[0009] In view of the above-mentioned deficiencies of the prior art, the present invention provides a method and system for identifying white peony varieties based on multimodal organ feature fusion. This method can integrate multi-organ information, support differentiated feature expression, and adaptively adjust the identification strategy according to modal quality, thereby solving the shortcomings of existing white peony variety identification technologies in cross-organ feature modeling and unstable modality processing.

[0010] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0011] The first aspect is a method for identifying white peony varieties based on multimodal organ feature fusion, which includes the following steps:

[0012] S1. Collect and preprocess root modal images and leaf and flower modal images of Paeonia lactiflora;

[0013] S2. Input the preprocessed root modality image and leaf / flower modality image into the deep learning backbone network containing the organ perception feature enhancement module. The organ perception feature enhancement module includes a structure enhancement branch, a color / texture enhancement branch, and a global context branch. The organ perception feature enhancement module generates organ-enhanced feature maps by performing conditional feature fusion by combining organ type embedding vectors and a gating fusion mechanism.

[0014] S3. Based on the enhanced feature maps of organs, construct independent classification sub-networks to predict varieties in root modality images and leaf and flower modality images, and generate corresponding classification probability vectors;

[0015] S4. Calculate the prediction uncertainty based on the classification probability vector; generate fusion weights based on the prediction uncertainty.

[0016] S5. Based on the fusion weight, the classification probability vectors of the root modality image and the leaf and flower modality image are weighted and fused to generate a fused prediction probability vector; the white peony variety is identified based on the fused prediction probability vector.

[0017] Preferably, step S1 includes:

[0018] The root modality image and the leaf / flower modality image are input into the bimodal image preprocessing module for preprocessing. The preprocessing of the root modality image includes one or more of the following: subject region extraction, size normalization, pixel value normalization, noise suppression, and conversion to tensors. The preprocessing of the leaf / flower modality image includes one or more of the following: subject region localization, size normalization, pixel value normalization, image enhancement and background weakening, and conversion to tensors. The root modality image and the leaf / flower modality image are kept consistent in size, brightness range, color space, and background complexity when input into the bimodal image preprocessing module.

[0019] Preferably, in step S2, the structure enhancement branch enhances the radial structure, main root and branch root arrangement, and main root and branch root thickness differences of the root modality image by using a multi-scale receptive field convolution operator, and enhances the leaf vein direction, leaf edge contour, and petal layering structure of the leaf and flower modality image. The color and texture enhancement branch enhances the color distribution and texture details of the root modality image and the leaf and flower modality image by using a color-texture attention operator based on a combination of channel attention and spatial attention. The global context branch generates a global semantic vector through global average pooling or global attention, and obtains global features through nonlinear transformation. The global context branch aims to obtain global semantic information of the entire input feature map, providing an overall reference for subsequent fusion decisions.

[0020] Preferably, step S2 further includes generating independent organ embedding vectors for the root modality image and the leaf and flower modality image respectively; the organ-gated fusion unit of the organ-aware feature enhancement module receives the outputs of the structure enhancement branch, the color and texture enhancement branch and the global context branch, and combines the organ embedding vectors and global features, and performs weighted summation of the features of each branch through a set of learnable gating weights, and then generates an organ-enhanced feature map through residual connection.

[0021] Preferably, step S4 includes:

[0022] Based on the classification probability vectors of the root mode and the leaf-flower mode, the uncertainty calculation unit is used to calculate the prediction uncertainty of each mode based on the degree of dispersion of the probability distribution, the inverse measure of the maximum probability value, or other statistical functions that reflect the concentration of probability.

[0023] Preferably, step S4 further includes:

[0024] Based on the prediction uncertainty, a weight generation unit is used to generate fusion weights for the root mode image and the leaf and flower mode image based on the inverse proportional function, linear normalization, or other mapping functions.

[0025] Preferably, the basis for generating the fusion weights includes image quality indicators; the weight generation mechanism is two-level or multi-level; and the fusion method includes at least one of linear weighting and nonlinear fusion strategies.

[0026] Preferably, step S4 further includes: using a multimodal dynamic fusion module to perform weighted fusion of the classification probability vectors of the root modality image and the leaf and flower modality image; the multimodal dynamic fusion module uses multiple uncertainty indicators to jointly measure the reliability of the root modality and leaf and flower modality predictions.

[0027] As preferred varieties, white peony varieties include Anhui Changxing Dahonghua, Anhui Pubang, Shandong Fenhua, Shandong Honghua, Sichuan double-flowered white peony, Sichuan double-flowered pink peony, Zhejiang Dahongpao, Zhejiang Gaogan, Zhejiang Chajiaogen, Zhejiang Kuanyefen, Zhejiang Wulong Tanhai, and Zhejiang Yanzhidianyue.

[0028] Secondly, the system employing the white peony variety identification method based on multimodal organ feature fusion as described above includes:

[0029] Image acquisition device for acquiring modal images of white peony root and leaf / flower;

[0030] The data processing device includes a dual-modal image preprocessing module, a deep learning backbone network with an organ perception feature enhancement module, and a multimodal dynamic fusion module; the organ perception feature enhancement module includes a structure enhancement branch, a color and texture enhancement branch, and a global context branch; the organ perception feature enhancement module performs conditional feature fusion by combining organ type embedding vectors and a gating fusion mechanism;

[0031] The storage device is used to save root modality images, leaf and flower modality images, and data generated during the deep learning recognition process, including various parameters, recognition logs, and historical recognition results.

[0032] The output device is used to output the recognition content, including the white peony variety name, prediction probability, and fusion weight.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] (1) Utilizing the differences of multiple organs to achieve more comprehensive phenotypic information expression. This invention introduces dual-modal image input of root modal image and leaf and flower modal image, and comprehensively utilizes complementary phenotypic information such as root topology, leaf vein texture and petal color and morphology, so that the model can obtain more comprehensive and discriminative feature expression than single organ image, thereby improving the sufficiency and reliability of variety identification.

[0035] (2) Achieving differentiated cross-organ expression capabilities through organ-sensing feature enhancement. The organ-sensing feature enhancement module OAE-Block designed in this invention achieves differentiated expression capabilities across organs through collaborative modeling of structural enhancement branches, color and texture enhancement branches, and global context branches, combined with organ type embedding and gating fusion mechanisms. This enables the network to adaptively adjust the degree of attention to structural or texture features based on whether the input is a root image or a leaf / flower image. Compared to conventional network structures that use a uniform convolutional branch for different organs, this invention can significantly improve the specificity and discriminative sensitivity of cross-organ feature expression.

[0036] (3) Improve the stability of multimodal discrimination through an uncertainty-driven dynamic fusion mechanism. The uncertainty-driven multimodal dynamic fusion module DUF proposed in this invention can adaptively adjust the contribution ratio of the root modality image and the leaf and flower modality image according to their prediction confidence. When the prediction uncertainty of a certain modality increases due to illumination, occlusion, or defocus, the system can automatically reduce its weight, thereby avoiding misjudgments caused by fixed weights or simple averaging under conditions of uneven modal quality. This mechanism significantly improves the stability and robustness of multimodal fusion in complex shooting environments.

[0037] (4) Unifying the input feature distribution through bimodal preprocessing enhances the consistency of the model across samples. The bimodal preprocessing module of this invention, through steps such as main region extraction, size normalization, pixel value normalization, and conversion to tensors, makes the root mode and leaf and flower mode images more consistent in terms of input size, pixel value, and background complexity reduction. This significantly reduces the impact of differences in acquisition equipment and environmental changes on feature extraction, providing a stable input foundation for subsequent deep feature learning, thereby improving the recognition consistency across shooting conditions and batches of samples.

[0038] (5) The method has a modular structure and strong scalability, and can be extended to the multi-organ identification of other plants. The method of this invention consists of modules such as preprocessing, organ perception feature enhancement, multimodal adaptive fusion, and classification prediction. The overall architecture is clear and the module interfaces are well-defined. The branch structure can be flexibly expanded or the fusion strategy can be adjusted according to the characteristics of different plant organs. Therefore, this method is not only suitable for the identification of white peony varieties, but can also be extended to other Chinese medicinal materials or plant species where the phenotypic information of multiple organs has an important impact on the identification accuracy. It has good scalability and application prospects. Attached Figure Description

[0039] To more clearly illustrate the technical solution of the present invention, the accompanying drawings are used to schematically demonstrate the structure and flow of the method and system described in the present invention. It should be understood that the drawings are only for illustrating the principles of the present invention and do not constitute any limitation thereof; unless otherwise specified, the reference numerals in the drawings can be used interchangeably. Those skilled in the art can understand the overall flow and key module composition of the present invention based on the accompanying drawings.

[0040] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0041] Figure 2 This is a schematic diagram of the dual-modal image preprocessing module of the present invention;

[0042] Figure 3 This is a schematic diagram of the organ sensing feature enhancement module (OAE-Block) of the present invention;

[0043] Figure 4 This is a schematic diagram of the multimodal dynamic fusion module (DUF) of the present invention;

[0044] Figure 5 This is a schematic diagram of the white peony variety identification system of Embodiment 1 of the present invention. Detailed Implementation

[0045] To make the technical means, inventive features, objectives, and effects of the invention readily understandable, the invention is further described below with reference to specific illustrations. However, the invention is not limited to the embodiments described below.

[0046] It should be noted that the structures, proportions, sizes, etc., illustrated in the accompanying drawings of this specification are only used to complement the content disclosed in the specification for those skilled in the art to understand and read, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0047] like Figure 1 As shown, this invention provides a method for identifying white peony varieties based on multimodal organ feature fusion, comprising the following steps:

[0048] S1. Image Acquisition and Dual-Modal Preprocessing: Images of peony root and leaf / flower parts were acquired separately. The raw images were then input into the dual-modal image preprocessing module. The root and leaf / flower modal images underwent processing including main region extraction, background removal or weakening, size normalization, pixel value normalization, and noise suppression, resulting in two types of standardized input images. This preprocessing ensures greater consistency in size, brightness, color space, and background complexity across different modalities, providing a stable and aligned input feature distribution for the subsequent organ feature enhancement module and reducing unstructured interference caused by differences in acquisition conditions.

[0049] S11. Image Acquisition: Acquire images of the roots and leaves or flowers of the white peony using an image acquisition device, which will serve as the root modality image and leaf / flower modality image, respectively. The acquisition method can be an industrial camera, mobile phone camera, digital camera, or other device with imaging capabilities, depending on the actual application scenario. The acquired images can be RGB images or other commonly used image formats. This step aims to provide independent but complementary information sources for subsequent multimodal processing.

[0050] S12. Dual-modal image preprocessing: The root modality image and the leaf / flower modality image are input into the dual-modal image preprocessing module. The module performs various processing steps on the two types of images, including subject region extraction, background weakening or removal, size normalization, pixel value normalization, and noise suppression. This module ensures that different modalities have consistent input specifications and higher stability before entering the feature extraction network, reducing the interference of acquisition differences on subsequent feature learning.

[0051] Combination Figure 2 The dual-modal image preprocessing module of the present invention will be described in detail below. This module constructs independent but stylistically consistent preprocessing flows for root modal images and leaf and flower modal images, aiming to standardize and structure images of different modalities before entering the neural network, so as to reduce interference caused by differences in image acquisition conditions and provide a unified and stable input feature distribution for subsequent organ perception feature enhancement.

[0052] Root modality image preprocessing: Root images often contain noise such as soil, excavation marks, shadows, or background debris, which can reduce the stability of feature extraction if directly input. Therefore, this embodiment employs one or more of the following steps for root modality images:

[0053] 1. Main region extraction: The main region of the root can be extracted using semantic segmentation models, object detection models or threshold segmentation, and the background region can be removed or weakened so that the model can focus on learning root structure information related to the variety.

[0054] 2. Size normalization: The main body area is scaled proportionally to the preset input size and centered to ensure that different samples are consistent in spatial scale, avoiding inconsistent response of the convolutional receptive field to key structures due to size differences;

[0055] 3. Pixel value normalization: Using methods such as brightness correction, histogram matching, or color space transformation to reduce lighting differences, making the brightness range and color distribution of images acquired in different batches more consistent;

[0056] 4. Noise Suppression: Suppress acquisition noise through bilateral filtering, guided filtering, or lightweight convolutional smoothing, enhance root bark texture and topology, and make it more suitable for feature learning of subsequent structure enhancement branches;

[0057] 5. Convert to Tensors: Convert the normalized image data into PyTorch tensor format for input into deep learning networks.

[0058] In this invention, the three essential preprocessing methods for root modality images are size normalization, pixel value normalization, and conversion to tensors, wherein:

[0059] Size normalization: The acquired root modality images are scaled proportionally to 224×224 pixels. Specifically, a bilinear interpolation algorithm (such as the A.Resize function in the Albumentations framework) is used to scale the original image from 2560×2560 pixels to 224×224 pixels, ensuring that different samples maintain a consistent spatial scale and avoiding inconsistent responses of the convolutional receptive field to key structures due to size differences.

[0060] Pixel value normalization: Mapping the scaled image pixel values ​​from the [0, 255] range to a standardized distribution. This involves three steps:

[0061] Step 1: Divide the pixel value by 255.0 and map it to the [0,1] range;

[0062] Step 2: Subtract the mean = [0.485, 0.456, 0.406] (RGB three channels);

[0063] Step 3: Divide by the standard deviation std = [0.229, 0.224, 0.225] (RGB three channels).

[0064] The mean and standard deviation mentioned above were obtained based on the ImageNet dataset. After normalization, the pixel value range is approximately [-2, 2], making the brightness range and color distribution of images acquired in different batches more consistent.

[0065] Convert to tensor: Convert the normalized image data into PyTorch tensor format with dimensions [3, 224, 224], where 3 represents the three RGB channels, so that it can be input into the deep learning network.

[0066] Of course, in addition to the necessary steps mentioned above, the root modality image preprocessing can optionally perform the following enhancement steps to improve the model robustness:

[0067] 1. Main region extraction: The main region of the root can be extracted using semantic segmentation models, object detection models or threshold segmentation, and the background region can be removed or weakened so that the model can focus on learning root structure information related to the variety.

[0068] 2. Noise Suppression: Acquisition noise can be suppressed through bilateral filtering, guided filtering, or lightweight convolutional smoothing, thereby enhancing root bark texture and topological structure to better suit the feature learning of subsequent structural enhancement branches.

[0069] Leaf and flower modal image preprocessing: Leaf and flower images have more color and texture variations, and the shooting background may be relatively complex. The following preprocessing methods are used:

[0070] 1. Main Region Localization: Use detection models or region localization algorithms to locate leaf or flower regions, and then crop them so that the network focuses on the texture and color regions related to the variety.

[0071] 2. Size normalization: The size of the cropped region is normalized to ensure that the input meets the resolution requirements of the network and reduces feature shift caused by differences in leaf size.

[0072] 3. Pixel value normalization: Using automatic white balance or color drift correction, the brightness and hue of key areas such as the green channel of leaves and the color of petals are kept consistent under different acquisition conditions;

[0073] 4. Image enhancement and background weakening: Fine-grained textures can be enhanced by sharpening and contrast enhancement, while non-subject backgrounds can be weakened by blurring or suppression strategies, making the vein texture and petal color distribution more prominent.

[0074] 5. Convert to Tensors: Convert the normalized image data into PyTorch tensor format for input into deep learning networks.

[0075] In this invention, the three essential preprocessing methods for leaf and flower modal images are size normalization, pixel value normalization, and conversion to tensors, wherein:

[0076] Size normalization: The acquired leaf and flower modal images are scaled proportionally to 224×224 pixels. Specifically, a bilinear interpolation algorithm (such as the A.Resize function in the Albumentations framework) is used to scale the original image from 640×640 pixels to 224×224 pixels, so that the input meets the resolution requirements of the network and reduces feature shift caused by differences in leaf size.

[0077] Pixel value normalization: The same normalization method as the root mode is used, consisting of three steps:

[0078] Step 1: Divide the pixel value by 255.0 and map it to the [0,1] range;

[0079] Step 2: Subtract the mean = [0.485, 0.456, 0.406] (RGB three channels);

[0080] Step 3: Divide by the standard deviation std = [0.229, 0.224, 0.225] (RGB three channels).

[0081] The mean and standard deviation mentioned above were obtained based on the ImageNet dataset. After normalization, the pixel values ​​ranged from approximately [-2, 2], ensuring that the brightness and hue of key areas such as the green channel of leaves and the color of petals remained consistent under different acquisition conditions.

[0082] Convert to tensor: Convert the normalized image data into PyTorch tensor format with dimensions [3, 224, 224], where 3 represents the three RGB channels, so that it can be input into the deep learning network.

[0083] Of course, in addition to the necessary steps mentioned above, the leaf and flower modality image preprocessing can optionally perform the following enhancement steps to improve the model robustness:

[0084] 1. Main Region Localization: Use detection models or region localization algorithms to locate leaf or flower regions, and then crop them so that the network focuses on the texture and color regions related to the variety.

[0085] 2. Image enhancement and background weakening: Fine-grained textures can be enhanced by sharpening and contrast enhancement, while non-subject backgrounds can be weakened by blurring or suppression strategies, making the vein texture and petal color distribution more prominent.

[0086] Bimodal Unified Specification: Although the root mode and leaf / flower mode can adopt different preprocessing strategies, in this invention, it is preferable to ensure that the two meet a unified specification before entering the deep learning network, including: consistent input size, consistent pixel values, and background complexity reduced to a similar level. The unified specification helps the subsequent OAE-Block to perform aligned feature enhancement in the same feature space and avoids cross-modal feature shift caused by differences in modal input.

[0087] Through the above preprocessing steps, this invention significantly improves the structural stability of the root modality image and leaf and flower modality image of peony, making the subsequent organ perception features more interpretable and effective, while providing a more consistent feature basis for multimodal dynamic fusion.

[0088] S2. Organ-Aware Feature Extraction: The preprocessed root modality image and leaf / flower modality image are input into a deep learning backbone network containing the OAE-Block. The OAE-Block consists of a structure enhancement branch, a color / texture enhancement branch, and a global context branch, and combines organ type embedding vectors with a gating fusion mechanism to achieve differentiated feature enhancement for different organ images. Unlike existing convolutional networks that use a uniform feature extraction method for different organs, this invention improves cross-organ feature expression capabilities by explicitly injecting organ semantics and using a gating structure for conditional feature fusion, resulting in enhanced organ feature maps.

[0089] In this invention, the deep learning backbone network is, for example, a deep convolutional neural network such as ResNet18, ResNet50, or VGG16.

[0090] In this invention, a ResNet18 network is used as an example. The preprocessed root modality image and leaf / flower modality image, converted to tensor format, are input into the ResNet18 backbone network. After passing through the ResNet18 Conv1 layer, Layer1 layer, Layer2 layer, and Layer3 layer, the output is a C×H×W input feature map X, where C is the number of channels, H is the height of the input feature map, and W is the width of the input feature map. The input feature map X is then input to the Organ-aware Enhancement Block (OAE-Block) for organ-aware feature enhancement. After organ-aware feature enhancement, it passes through Layer4 to obtain the two modality feature vectors. For example, after both modal images are converted to tensor format of [3, 224, 224] dimensions, they are input into the ResNet18 backbone network. After passing through the Conv1, Layer1, and Layer2 layers of ResNet18, the size of the input feature map X becomes 128×28×28 (128 channels, 28 height, 28 width). After passing through Layer3, the size of the input feature map X becomes 256×14×14 (256 channels, 14 height, 14 width). At this point, the input feature map X is input into OAE-Block for organ-aware feature enhancement. After organ-aware feature enhancement, it passes through Layer4 and global average pooling to obtain the two modal feature vectors.

[0091] Combination Figure 3This paper provides a detailed description of the organ-aware enhancement module (OAE-Block) described in this invention. OAE-Block is embedded in the intermediate feature layer of the deep learning network backbone. It is used to adaptively enhance features based on the differences in structural morphology, color texture, and semantic information between the root modality image and the leaf and flower modality image of Paeonia lactiflora, thereby improving the specificity and discriminative ability of cross-organ feature expression.

[0092] like Figure 3 As shown, OAE-Block includes at least a structure enhancement branch, a color and texture enhancement branch, a global context branch, and an organ-gated fusion unit. Each branch is based on the input feature map. The process operates in parallel, ultimately generating organ enhancement feature maps through a fusion unit. The following describes each component:

[0093] S21, Structural Enhancement Branch: This branch is used to enhance information related to the morphological structure of Paeonia lactiflora organs. For root modality images, the structural enhancement branch can highlight the topological structure, the arrangement and thickness differences of the main root and lateral roots; for leaf and flower modality images, this branch can be used to enhance the direction of leaf veins, the outline of leaf margins, and the layered structure of petals. This embodiment can use multi-scale convolution, dilated convolution (dilated convolution), or other convolution methods with multi-scale receptive fields to model the input feature map to fully capture local and global structural features.

[0094] In this invention, the calculation form of the structural reinforcement branch is written as: .in, This represents a convolution operator with multi-scale receptive fields. This branch enhances the feature map output of the structural enhancement branch. Through this branch, the network becomes more sensitive to the main root and lateral root structures in the root mode, and more sensitive to leaf veins and contour shapes in the leaf and flower mode.

[0095] The convolution operators for the multi-scale receptive field specifically include: a convolution operator with a kernel size of 3×3, a convolution operator with a kernel size of 5×5, a convolution operator with a kernel size of 7×7, and an expanded convolution operator with an expansion rate of 2 (with a kernel size of 3×3).

[0096] The specific calculation steps of the convolution operator for the multi-scale receptive field are as follows:

[0097] Step 1: Perform convolution operation on the input feature map using a 3×3 convolution kernel, with padding of 1, and output feature map F1;

[0098] Step 2: Perform convolution operation on the input feature map using a 5×5 kernel, with padding of 2, and output feature map F2;

[0099] Step 3: Perform convolution operation on the input feature map using a 7×7 kernel, with padding of 3, and output feature map F3;

[0100] Step 4: Perform a 3×3 dilated convolution operation on the input feature map with a dilation rate of 2 and a padding of 2, and output feature map F4;

[0101] Step 5: Concatenate feature maps F1, F2, F3, and F4 along the channel dimension to obtain the concatenated feature map F_concat;

[0102] Step 6: Perform channel integration on F_concat using 1×1 convolution to output the final structure-enhanced feature map Fs.

[0103] Among them, the 3×3 convolution operator is used to capture local detail features, the 5×5 convolution operator is used to capture medium-scale structures, the 7×7 convolution operator is used to capture large-scale morphology, and the dilated convolution operator with a dilation rate of 2 is used to expand the receptive field without significantly increasing the computational cost. The feature results obtained from convolution operators of different scales are fused by concatenation and further integrated by channel dimension using a 1×1 convolution operator to obtain the feature representation for subsequent processing.

[0104] S22, Color and Texture Enhancement Branch: This branch focuses on enhancing the color distribution and texture details of the organs in Paeonia lactiflora. For example, the fine texture on the leaf surface, the color banding of the petals, and the color gradations of the root bark all exhibit certain varietal differences. The Color and Texture Enhancement Branch can use a combination of channel attention and spatial attention to weighted enhance channels or spatial locations related to color and texture, enabling the network to achieve higher responses in key texture regions.

[0105] In this invention, the output of the color texture enhancement branch is represented as: .

[0106] in, This represents a color-texture attention operator based on a combination of channel attention and spatial attention. This branch enhances the color and texture feature maps. Compared to existing methods that only use a uniform convolutional structure, this branch explicitly enhances variety-related color and texture information.

[0107] Specifically, the calculation steps for channel attention are as follows: First, perform global average pooling on the input feature map X in the spatial dimension, compressing X into a C×1×1 vector. Second, compress the number of channels from C to C / 16 (compression ratio of 16) using a 1×1 convolution, and then apply the ReLU activation function. Third, restore the number of channels from C / 16 to C using another 1×1 convolution, and finally use the Sigmoid function to map the values ​​to the [0,1] interval to obtain the channel weight vector. Each element of this channel weight vector represents the importance of the corresponding channel.

[0108] The specific steps for calculating spatial attention are as follows: A 7×7 convolution kernel is used to convolve the input feature map X, mapping C channels to one channel. Then, the Sigmoid function is used to map the values ​​to the [0,1] interval, resulting in a spatial weight vector. Each position in this spatial weight vector represents the importance of that spatial location.

[0109] The channel weight vector and spatial weight vector are used to weight the original features channel-wise and spatially, respectively. That is, the original features are first multiplied by the channel weight vector, and then the result is multiplied by the spatial weight vector.

[0110] S23, Global Context Branch: This branch aims to obtain global semantic information from the entire input feature map X, providing an overall reference for subsequent fusion decisions. The global context branch can generate a global semantic vector through global average pooling, global attention, or other feature compression methods, and obtain discriminative global features through nonlinear transformation. In OAE-Block, global semantic features are mainly used to guide the weight generation of organ-gated fusion units, rather than directly participating in convolution as independent channels.

[0111] Symbol explanation: Represents the set of real numbers. express A 3D real vector space. For example, express yes A real tensor of dimension 1.

[0112] In this invention, the following form is adopted: , Wherein, GAP(X) represents global average pooling of the input feature map X in the spatial dimension, outputting a global semantic description vector. . Represents the global vector Multilayer perceptron (fully connected network) for nonlinear mapping. This is the initial global semantic description vector. . This is the transformed global semantic vector. .

[0113] In this invention, the specific calculation process of the global context branch is as follows: using the input feature map... For example, where C is the number of channels, and H and W are the height and width of the feature map, respectively. Step 1: Perform global average pooling on the input feature map X in the spatial dimension, compressing the input feature map X from C×H×W to a C×1×1 global semantic description vector. ,in It is a C-dimensional global semantic description vector. Representation of feature map In spatial location The value at point (containing all C channels); Step 2: Convert the C×1×1 three-dimensional tensor Flattened into a one-dimensional vector The dimensions remain unchanged. Step 3: Perform a nonlinear transformation using the first fully connected FC1 and ReLU activation functions to reduce the dimension from C to [value missing]. , ,in , The fourth step involves using a second fully connected FC2 layer to increase the dimension from hidden_dim to C, resulting in the final global semantic vector. FC2 is a fully connected layer that maps the hidden_dim dimensional vector to a C dimensional vector. This is the final global semantic vector, used to guide subsequent fusion decisions. Among them, GAP, FC1, ReLU, and FC2 together constitute a nonlinear transformation layer in the form of a multi-layer perceptron (MLP).

[0114] The output of the global context branch is represented as follows: Here, Expand represents the spatial expansion operation, which expands the global semantic vector. Expanded into a global feature map By Each channel value is broadcast and copied to Spatial dimension allows each spatial location to obtain the same global context information. For the global context feature map, and the output of the structure enhancement branch and the output of the color and texture enhancement branch They are input together into the organ-gated fusion unit.

[0115] S24. Organ Type Embedding Vector: To enable the feature enhancement process to differentiate based on the organ type corresponding to the input image, this embodiment sets organ type embedding vectors for the root modality image and the leaf / flower modality image respectively. The organ type embedding vector is generated in a learnable manner and represents the differences in structural morphology, texture features, and discriminative information sources among different organs. It is introduced as prior information into the organ-gated fusion unit to guide the fusion strategy of multi-branch features. During training, a labeled white peony variety dataset is used. The cross-entropy loss function and backpropagation algorithm are employed to optimize the organ type embedding vector. The optimization of the organ type embedding vector is existing technology in this field and will not be elaborated here.

[0116] In this invention, organ type can be determined by discrete labels. Indicate, for example or Organ type embedding can be achieved through a learnable embedding layer, which inputs discrete labels representing modality categories into the embedding layer and maps them to organ embedding vectors of fixed dimensions. ,in, Represents an embedding map. This refers to the organ embedding vector. The organ embedding vector is used in the generation process of gating weights. By introducing organ type embedding, the organ gating fusion unit is able to adjust features based on organ differences under a unified network structure, thereby achieving an adaptive balance between structure-dominated features and color / texture-dominated features. This is beneficial for improving the stability and discriminative power of feature representation in cross-organ variety recognition tasks.

[0117] In this invention, the organ type embedding vector is generated as follows: a learnable vector parameter with a dimension of 64 is initialized for both the root modality image and the leaf / flower modality image. Specifically, the organ embedding vector of the root modality is represented as follows: The organ embedding vector of the leaf-flower modality is represented as These two vectors are randomly initialized and automatically learned through backpropagation during the training of the deep learning network. They represent the differences in structural morphology, texture features, and sources of discriminative information among different organs. In practical use, the corresponding organ embedding vector is selected based on the modality type of the input image: if the input is a root modality image, then... If the input is a leaf and flower modality image, then select... .

[0118] The gated fusion unit generates fusion weights for each branch using a multilayer perceptron and fuses the features of each branch using a weighted summation method. Specifically, the gated fusion unit receives a global semantic vector and an organ embedding vector as input, concatenates them, inputs the concatenation into the multilayer perceptron, and outputs three fusion weights, corresponding to the structure enhancement branch, color and texture enhancement branch, and global context branch, respectively. Finally, the Softmax function is used to normalize the three weights so that their sum is 1.

[0119] S25, Organ-Gated Fusion Unit: This unit receives the outputs of the structure enhancement branch, color and texture enhancement branch, and global context branch. It combines the organ embedding vector with global semantic features and performs a weighted summation of the features from each branch using a set of learnable gating weights. These weights can be generated using a multilayer perceptron, tensor fusion, or attention structure, allowing for differentiation in feature fusion strategies between the root modality image and the leaf / flower modality image. For example, for the root modality, the gating mechanism can favor the structure enhancement branch; for the leaf / flower modality, it can favor the color and texture enhancement branch. Finally, the fused features are added to the input features via residual connections to obtain a stable and organ-sensitive enhanced feature map.

[0120] In this invention, the generation form of the gating weight is written as: ,in, Represents the global semantic vector With organ embedding vector splicing, This represents a multilayer perceptron used to generate weights. Assign weights to the corresponding structural enhancement branch, color / texture enhancement branch, and global context branch, and satisfy... .

[0121] Based on this, the fusion characteristics of the three branches are represented as follows: ,in, Represented by global semantic vector The global feature map is obtained through linear mapping and expansion. Finally, the output of OAE-Block is obtained through residual connections: ,in, This involves enhancing the feature map for organs. Compared to existing structures that do not distinguish organ semantics and simply perform fixed weighting or simple addition of multi-branch outputs, this embodiment introduces organ embedding vectors. With conditional gating weights This allows the root mode and leaf / flower mode to exhibit learnable differences in feature fusion strategies, thus constituting an improvement of the OAE-Block structure in this invention.

[0122] Through the above structure, OAE-Block can adaptively select a feature representation method that is more suitable for the organ corresponding to the input image, thereby improving the effectiveness of overall feature extraction and providing a more reliable representation for subsequent classification prediction and intermodal fusion.

[0123] Following OAE-Block, continue using Layer 4 of the ResNet18 deep learning backbone network. The feature map is further extracted as Feature map; a global average pooling layer is used to... The feature map is pooled into a 512-dimensional vector.

[0124] S3. Intramodal Classification Prediction: Independent classification sub-networks are constructed based on the enhanced organ feature maps to predict the variety of root modality images and leaf / flower modality images, obtaining the corresponding classification probability vectors. Unlike existing methods that directly stack convolutional layers or use a unified classification head, the classification sub-networks of this invention are structurally matched to the OAE-Block output, fully utilizing the enhanced organ features after the three branches, making the classification process more sensitive to differences in organ features, thereby improving the accuracy of intramodal prediction.

[0125] Specifically, the classification sub-network includes the following structure:

[0126] (1) Fully connected classification layer: maps the 512-dimensional vector to 12 categories (corresponding to 12 varieties of white peony), and outputs a 12-dimensional logits vector. W is the weight matrix, and b is the bias vector;

[0127] (2) Softmax normalization layer: converts the 12-dimensional logits vector into a 12-dimensional probability distribution vector.

[0128] The root modality classification subnetwork and the leaf-flower modality classification subnetwork adopt the same structure as described above, but their parameters are independent. The fully connected classification layer and the Softmax normalization layer in the two classification subnetworks are called the root modality classifier and the leaf-flower modality classifier, respectively. The parameters of the classification subnetworks (including the weights and biases of the fully connected layers) are automatically learned through the deep learning network training process. During training, a labeled white peony variety dataset is used, and the parameters are optimized using the cross-entropy loss function and the backpropagation algorithm. The parameter optimization part is a prior art in this field and will not be described in detail here.

[0129] In this invention, the 12 white peony varieties are Anhui Changxing Dahonghua, Anhui Pubang, Shandong Fenhua, Shandong Honghua, Sichuan Double-flowered White Flower, Sichuan Double-flowered Pink Flower, Zhejiang Dahongpao, Zhejiang Gaogan, Zhejiang Chajiaogen, Zhejiang Kuanyefen, Zhejiang Wulong Tanhai, and Zhejiang Yanzhidianyue.

[0130] S4. Based on the classification probability vectors obtained from intra-modal classification predictions, the uncertainty of the prediction results for the root mode and the leaf / flower mode is measured, and corresponding mode fusion weights are generated accordingly. Uncertainty reflects the reliability of the prediction results for each mode; modes with lower uncertainty are assigned higher weights during the fusion process. In one embodiment, uncertainty can be calculated based on the concentration of the classification probability distribution. The weight generation unit normalizes and adjusts the contribution ratio of each mode according to the uncertainty result, thereby enabling the multi-modal fusion process to adaptively adjust the prediction reliability for different samples, thus improving the stability of the overall recognition result.

[0131] S41. Uncertainty Calculation Unit: This unit receives the classification probability vectors of the root modality image and the leaf / flower modality image, and measures the prediction uncertainty of each modality using an index reflecting the concentration of the probability distribution. Higher uncertainty indicates that the prediction results are scattered and have lower reliability, while lower uncertainty indicates that the prediction results for that modality are more reliable. Let the probability vector of the root modality image be... K represents the number of varieties, and the probability vector of the leaf-flower modality image is... K represents the number of varieties. Uncertainty can be calculated using one of the following methods:

[0132] ① Based on the degree of dispersion of the probability distribution, for example, uncertainty is higher when the probability distribution is close to a uniform distribution, and lower when the probability is highly concentrated. This degree of dispersion can be quantified using an entropy function, for example: { , , i=1,2,3,……,K}, where K is the number of varieties.

[0133] ② Inverse measure based on the maximum probability value; for example, a larger maximum probability value indicates higher predictive reliability for that mode. The corresponding inverse uncertainty can be expressed as: , .

[0134] ③ Based on other statistical functions reflecting the concentration of probability. This invention does not limit the method of uncertainty calculation, as long as it can reflect the reliability of the probability vector. The entropy method and the maximum probability inverse method mentioned above can be regarded as examples of this type of method.

[0135] In this invention, the entropy function is specifically employed. , The degree of dispersion is quantified. The larger the entropy value, the closer the probability distribution is to a uniform distribution and the higher the uncertainty; the smaller the entropy value, the more concentrated the probability distribution and the lower the uncertainty.

[0136] S42, Weight Generation Unit: This unit generates fusion weights for the root mode and the leaf / flower mode based on the uncertainty calculation results. Generally, modes with lower uncertainty should be assigned higher fusion weights. In this embodiment, normalized weights for the two modes can be generated using an inverse proportional function, linear normalization, or other mapping functions, ensuring the sum of the weight values ​​is 1, thus facilitating subsequent weighted fusion.

[0137] For ease of understanding, let the uncertainties corresponding to the root mode and the leaf-flower mode be respectively... and In this invention, the weights are generated using the following common inverse normalization method: ;in and Let represent the fusion weights of the root mode and the leaf / flower mode, respectively, and satisfy the following: .

[0138] The weight generation method described above is consistent with the present invention's expectation that "the higher the reliability, the greater the weight." However, the present invention does not limit the specific normalization form and can flexibly choose according to actual application.

[0139] S5. Multimodal Dynamic Fusion: Based on the generated fusion weights, the classification probability vectors of the root mode and the leaf-flower mode are weighted and fused to obtain the final fused prediction probability vector. The white peony variety identification result is output according to the category corresponding to the highest probability. The fusion method of this invention can adaptively suppress the interference of low-quality modes, making the identification results more robust under complex acquisition conditions.

[0140] Combination Figure 4 This paper provides a detailed description of the Dynamic Uncertainty-aware Fusion (DUF) module of the present invention. This module is used to adaptively fuse the classification prediction results of the root mode and the leaf and flower mode at the decision layer, so that the final recognition result can be dynamically adjusted according to the prediction reliability of different modes, thereby maintaining stable and reliable recognition performance under complex acquisition conditions.

[0141] Multimodal dynamic fusion module: This unit performs weighted fusion of the classification probability vectors of the root mode and the leaf-flower mode to obtain the final fusion probability vector. For each category, its fusion probability is composed of the weighted sum of the probabilities of the two modes in the corresponding category. Subsequently, based on the category corresponding to the highest probability in the fusion probability vector, the final white peony variety identification result is output. The fusion method of this invention can adaptively adjust the modal contribution according to the difference in prediction confidence, so that the overall recognition performance can still be maintained even when the image quality of a certain modality is poor or the prediction is unstable.

[0142] For ease of explanation, let the classification probability vector of the root mode be... The classification probability vector for the leaf and flower modality is: The corresponding fusion weight is and Therefore, in this invention, the final fusion probability vector is represented as: ;in The dimension is equal to the number of variety categories. , its first Each component represents the fusion prediction probability for that category. However, this invention does not limit the fusion strategy; any weight-based differentiable weighting method can be used to implement this function.

[0143] Optional variations: In other implementations, the DUF module may also include the following improvements:

[0144] 1. Introduce multiple uncertainty indicators for joint measurement to more comprehensively reflect the reliability of predictions across different modes. For example, entropy-based uncertainty indicators can be labeled as follows: The uncertainty index based on the maximum probability inverse measure is defined as: Both can together constitute the uncertainty characteristic. A joint uncertainty quantity can be obtained through any learnable or fixed combination function f: ;in, , For the first The predicted probability of a class. This is a value describing the uncertainty after synthesis. This invention describes the function... The specific form is not limited and can be a linear combination, nonlinear mapping, or attention weighting, etc.

[0145] 2. Employ a two-level or multi-level weight generation mechanism so that the weights not only depend on uncertainty, but also on image quality indicators or other statistical features such as blur, brightness shift, exposure status, etc., to further enhance the adaptability of the weights.

[0146] 3. The fusion method can be extended from linear weighting to nonlinear fusion strategies, such as maximum value fusion, gated fusion, or attention-based fusion. Nonlinear fusion can be expressed as: ; where Fusion represents any fusion function that can achieve weighted or selective combinations.

[0147] Through the above optional variations, the DUF module of the present invention can be extended according to the complexity of the actual scenario, further improving the flexibility and robustness of multimodal fusion.

[0148] Through the above structure, the DUF module can achieve a dynamic balance of cross-modal prediction results, ensuring that the final recognition results remain highly robust under diverse shooting conditions and variations in sample quality. This module works in conjunction with OAE-Block to significantly enhance the stability and practicality of the overall recognition method.

[0149] Combination Figure 5 The present invention also discloses a white peony variety identification system based on multimodal organ feature fusion. This system can be used to deploy the method of the present invention to realize the automatic acquisition, processing, identification and output of white peony sample images. It is suitable for various application scenarios such as laboratory testing, primary screening at the place of origin and quality control in the processing workshop.

[0150] Image acquisition device: Used to acquire modal images of white peony roots and leaf and flower patterns. It can employ industrial cameras, mobile phone cameras, portable acquisition terminals, or other devices with imaging capabilities. The acquisition device can be configured with a fixed bracket, supplementary lighting, or a background board as needed to improve image quality and consistency.

[0151] The acquired images can be transmitted to the data processing device via wired or wireless means. To ensure consistency of input specifications, in one embodiment, the image acquisition device can also preset resolution, white balance, or exposure strategies to make the acquired data more suitable for the subsequent dual-modal preprocessing module.

[0152] Data processing device: This is the core component of the system, including a processor and a memory. The memory pre-stores program instructions and required model parameters for executing the method of this invention. When the processor executes these program instructions, it performs the following functions:

[0153] 1. Receive root mode images and leaf and flower mode images transmitted by the image acquisition device;

[0154] 2. Call the dual-modal image preprocessing module to perform subject extraction, size normalization, pixel value normalization, and conversion to tensors on the two types of images respectively;

[0155] 3. Call the deep learning model containing OAE-Block to perform organ perception feature enhancement and intra-modal classification prediction on the preprocessed image;

[0156] 4. Call the Multimodal Dynamic Fusion (DUF) module to generate fusion weights based on the prediction uncertainty of each modality, and output the final recognition result;

[0157] 5. Store or output the recognition results to the specified device or interface.

[0158] Storage device: Used to store data such as peony sample images, preprocessing parameters, deep learning model parameters, recognition logs, and historical recognition results. The storage device can be a local hard drive, solid-state drive, cloud storage service, or other read / write media.

[0159] In extended embodiments, the storage device can also maintain training and validation set data to support subsequent model updates or parameter fine-tuning, enabling the system to evolve over a long period of time.

[0160] Output device: Used to present the system's recognition results, which can be a display screen, mobile terminal interface, host computer software, or data interface. Output content includes the identified white peony variety name, predicted probability, fusion weight, and other information.

[0161] To adapt to industrial scenarios, the output device can further output the recognition results in JSON, table or report format, and can upload them to the traceability system, data management platform or enterprise quality control system through network interface.

[0162] Optional deployment methods: This system can be deployed on different hardware platforms according to specific needs, including:

[0163] 1. Local workstation deployment: Suitable for research scenarios and high-performance offline recognition, supporting large-scale batch processing;

[0164] 2. Edge computing device deployment: Suitable for environments with high real-time requirements, such as on-site sourcing and processing, reducing network dependence;

[0165] 3. Cloud server deployment: Suitable for large-scale sample batch identification, or as a unified service interface to connect with different production systems.

[0166] The three deployment methods described above can be used independently or hybridized through the modular software architecture provided by this invention, giving the system good portability and scalability. The above embodiments of this invention are intended to illustrate the technical solutions of this invention and are not intended to limit the invention. Those skilled in the art can make various modifications, substitutions, or optimizations to the step sequence, module structure, algorithm form, and parameter settings of this invention without departing from the spirit and substance of this invention, and all of these modifications and substitutions fall within the protection scope of this invention.

[0167] Regarding optional variations of the dual-modal image preprocessing module: In different image acquisition scenarios, other types of background weakening strategies, color unification methods, local region enhancement methods, or lightweight denoising algorithms can be adopted; the main body region extraction method can also be replaced with traditional image segmentation methods, clustering algorithms, edge detection, or threshold-based segmentation methods as needed. Furthermore, the order of preprocessing steps can be appropriately adjusted according to the business scenario, such as performing color correction before size normalization. All the above adjustments do not affect the essence of this invention.

[0168] Regarding optional variations of OAE-Block: the structural enhancement branch and the color / texture enhancement branch can employ convolutional kernels of different sizes, multi-scale feature fusion methods, or other attention structures; the global context branch can be replaced with a Transformer-based self-attention structure, pyramid pooling structure, or other modules with global information modeling capabilities; the generation method of organ embedding vectors can be based on learnable parameters, lookup tables, or other encoding forms, such as one-hot encoding or conditional vector generation networks; the gating fusion method can employ multilayer perceptrons, tensor fusion methods, channel-wise attention weighting, or other dynamic weight generation strategies. Any solution that can achieve feature enhancement based on organ type is considered an equivalent solution of this invention.

[0169] Regarding optional variations of the DUF module: the uncertainty measurement method can adopt methods based on distribution entropy, probability central tendency, probability variance, maximum probability inverse measure, or other statistics; the weight generation method can be linear normalization, Softmax normalization, inverse proportional mapping, or conditional attention mechanism; the fusion method can be extended to hierarchical fusion, gated fusion, attention-based nonlinear fusion strategy, or thresholding of classification probabilities based on fusion weights, etc. This invention does not limit the specific implementation form of the DUF module, as long as it satisfies the purpose of dynamically adjusting the fusion strategy according to modality credibility.

[0170] Regarding the optional variations of system deployment: the system can be deployed on local servers, mobile devices, edge computing nodes, or cloud clusters; it can also be used in conjunction with medicinal material traceability systems, origin management systems, or quality monitoring platforms according to actual business needs to achieve batch monitoring, automated quality control, or visual display functions.

[0171] In an extended implementation, the system can also be linked with data acquisition terminals, intelligent warehousing systems, or production lines to achieve an overall process of automatic data acquisition, automatic identification, and automatic recording, thereby improving overall production efficiency.

[0172] In summary, the methods and systems provided by this invention have good scalability and engineering adaptability. Although this specification has described the invention in detail with reference to the accompanying drawings, those skilled in the art will understand that various equivalent modifications and improvements made to it to meet specific needs fall within the protection scope of this invention.

[0173] Example 1:

[0174] This implementation is an example of a white peony variety identification process based on the method of the present invention.

[0175] This application example uses the identification of the Zhejiang Dahongpao tea variety as an example to detail the complete processing from a dual-modal input image to the final recognition result. In this example, uncertainty calculation adopts an entropy function-based method, and weight generation adopts an inverse proportional normalization method. Experiments have verified that this combination is the optimal configuration.

[0176] In this embodiment, the method for identifying white peony varieties based on multimodal organ feature fusion includes the following steps:

[0177] Step S1: Image Acquisition and Dual-Modal Preprocessing

[0178] S11. Image Acquisition: Input the root modal image and leaf-flower modal image of the peony to be identified. The image is an acquired plant organ image, and the image is a color image containing multiple pixels. In this embodiment, the input is the root modal image of the Zhejiang Dahongpao variety, with a size of 2560×2560 pixels and RGB three channels; the input is the leaf-flower modal image, with a size of 640×640 pixels and RGB three channels.

[0179] S12. Bimodal Image Preprocessing: The input bimodal images are preprocessed, including image resizing and pixel value normalization, to obtain standardized image data that meets the network input requirements. In this embodiment, the root modality image is scaled to 224×224 pixels using bilinear interpolation, and the leaf / flower modality image is scaled to 224×224 pixels using bilinear interpolation. Then, normalization is applied to both modal images, mapping pixel values ​​from [0,255] to a standardized distribution of approximately [-2,2]. The normalization parameters are: mean = [0.485, 0.456, 0.406], standard deviation std = [0.229, 0.224, 0.225]. After preprocessing, both modal images are converted to a tensor format of [3, 224, 224] dimensions.

[0180] S2. Organ perception feature extraction: The preprocessed image is input into a deep learning backbone network containing an organ perception feature enhancement module. The image is then subjected to feature extraction and enhancement through multi-scale convolution operators and attention mechanisms to obtain a deep feature representation of the image.

[0181] The root modality image in tensor format is input into the ResNet18 backbone network. After passing through the Conv1, Layer1, and Layer2 layers of the ResNet18 backbone network, the input feature map size becomes 128×28×28 (128 channels, 28 height, 28 width). After passing through Layer3, the input feature map size becomes 256×14×14 (256 channels, 14 height, 14 width). At this point, the input feature map X is input into OAE-Block for organ-aware feature enhancement, including structural enhancement branches, color and texture enhancement branches, global context branches, organ type embedding vectors, and gated fusion mechanisms. The output organ-enhanced feature map has a size of 256×14×14. After passing through Layer4 and global average pooling, a 512-dimensional root modality feature vector is obtained. The leaf and flower modality image in tensor format is processed through the same network structure, and after OAE-Block enhancement, Layer4, and global average pooling, a 512-dimensional leaf and flower modality feature vector is obtained.

[0182] In this invention, the organ type embedding vector is generated as follows: a learnable vector parameter with a dimension of 64 is initialized for both the root modality image and the leaf / flower modality image. Specifically, the organ embedding vector of the root modality is represented as follows: The organ embedding vector of the leaf-flower modality is represented as These two vectors were randomly initialized and automatically learned through backpropagation during the training of the deep learning network. A labeled dataset of white peony varieties was used during training. The vector parameters were optimized using the cross-entropy loss function and backpropagation. The vector parameter optimization is a prior art technique and will not be elaborated here. The mathematical expression for the organ embedding vector is: , .in Indicates the root modality label. This represents the leaf / flower modality label, and `Emb` is the embedding layer mapping function. In practical use, the corresponding organ embedding vector is selected based on the modality type of the input image: if the input is a root modality image, then... If the input is a leaf and flower modality image, then select... .

[0183] S3. Intramodal classification prediction: Based on the deep feature representation, namely the root modal feature vector and the leaf and flower modal feature vector, intramodal classification prediction is performed respectively, and the predicted probability value corresponding to each target category is output.

[0184] Both the root modality classifier and the leaf-flower modality classifier consist of a fully connected classification layer and a Softmax normalization layer. Specifically: the fully connected classification layer maps the 512-dimensional feature vector to 12 dimensions (corresponding to 12 varieties of white peony), using the following formula: Where W is The weight matrix is ​​denoted by , where b is a 12-dimensional bias vector; the Softmax normalization layer converts the 12-dimensional logits vector into a probability distribution vector, as shown in the formula. .

[0185] The root modality classifier and the leaf-flower modality classifier use the same network structure, but their parameters are independent (i.e., they have independent weight matrices W and bias vectors b). These parameters are automatically learned through the deep learning network training process. During training, a labeled white peony variety dataset is used, and the parameters are optimized using the cross-entropy loss function and the backpropagation algorithm. The parameter optimization part is a prior art technique and will not be elaborated here.

[0186] In this invention, the fully connected classification layer formula obtained by the root modality classifier is automatically learned through the deep learning network training process. ,in Here is the root mode weight matrix. The root mode bias vector is given; the formula for the fully connected classification layer obtained by the leaf and flower mode classifier is: ,in The leaf and flower mode weight matrix is ​​shown below. This represents the leaf-flower mode bias vector. Its specific value is automatically optimized during training, taking into account the training process and initialization method.

[0187] In this embodiment, the 12 white peony varieties are: Anhui Changxing Dahonghua (variety 1), Anhui Pubang (variety 2), Shandong Fenhua (variety 3), Shandong Honghua (variety 4), Sichuan Double-flowered White Flower (variety 5), Sichuan Double-flowered Pink Flower (variety 6), Zhejiang Dahongpao (variety 7), Zhejiang Gaogan (variety 8), Zhejiang Chajiaogen (variety 9), Zhejiang Kuanyefen (variety 10), Zhejiang Wulong Tanhai (variety 11), and Zhejiang Yanzhidianyue (variety 12).

[0188] In this embodiment, the 512-dimensional feature vector of the root mode is input into the root mode classifier, which outputs the predicted probabilities of 12 white peony varieties, specifically [0.05, 0.08, 0.12, 0.15, 0.25, 0.10, 0.08, 0.06, 0.05, 0.03, 0.02, 0.01]. The fifth value, 0.25, is the largest and corresponds to the Zhejiang Dahongpao variety. The 512-dimensional feature vector of the leaf-flower modality is input into the leaf-flower modality classifier, which outputs the predicted probabilities of 12 varieties, specifically [0.03, 0.05, 0.08, 0.10, 0.18, 0.12, 0.15, 0.10, 0.08, 0.06, 0.03, 0.02]. The fifth value, 0.18, is the largest and also corresponds to the Zhejiang Dahongpao variety.

[0189] S4. Calculate the prediction uncertainty of each mode based on the predicted probability values, and generate fusion weights based on the uncertainties. In this embodiment, an uncertainty calculation method based on the entropy function is used to calculate the entropy value of the root mode probability distribution vector to obtain the root mode uncertainty. Calculate the entropy value of the leaf-flower modal probability distribution vector to obtain the leaf-flower modal uncertainty. Then, the inverse proportional normalization method is used to generate the fusion weights: after normalization, the root mode fusion weights are obtained. Leaf and flower modal fusion weights ,satisfy .

[0190] S5. The classification prediction results of the root mode and leaf-flower mode are weighted and fused according to the fusion weight to obtain the final fusion prediction probability value. In this embodiment, the prediction probabilities of 12 varieties are weighted and fused separately. Taking the Zhejiang Dahongpao variety (ranked 5th among the 12 varieties) as an example, the fusion probability is 0.525×0.25 + 0.475×0.18 = 0.217. Similarly, the fusion probabilities of the other 11 varieties are calculated, and the final fusion prediction probability vector is obtained [0.042, 0.067, 0.102, 0.128, 0.217, 0.109, 0.113, 0.078, 0.064, 0.044, 0.024, 0.014].

[0191] S6. Determine the final recognition result based on the fused prediction probability values. The category with the highest prediction probability value is taken as the recognition category corresponding to the input image. In the fused prediction probability vector, the 5th value is the largest, 0.217. Therefore, the recognition result is Zhejiang Dahongpao variety, with a confidence level of 21.7%.

[0192] The above steps achieve a complete processing flow from a dual-modal input image to the final recognition result.

[0193] Example 2

[0194] In this embodiment, to verify the effectiveness of the method of the present invention, the variety of white peony was detected based on 283 white peony samples collected in practice.

[0195] Step S1: Image Acquisition and Dual-Modal Preprocessing

[0196] S11. 283 root modal images (2560×2560 pixels, RGB three-channel) and 283 leaf and flower modal images (640×640 pixels, RGB three-channel) were collected. The varieties are: Anhui Changxing Dahonghua (variety 1), Anhui Pubang (variety 2), Shandong Fenhua (variety 3), Shandong Honghua (variety 4), Sichuan Double-flowered White Flower (variety 5), Sichuan Double-flowered Pink Flower (variety 6), Zhejiang Dahongpao (variety 7), Zhejiang Gaogan (variety 8), Zhejiang Kaoguagen (variety 9), Zhejiang Kuanyefen (variety 10), Zhejiang Wulong Tanhai (variety 11), and Zhejiang Yanzhidianyue (variety 12).

[0197] S12. The root modality image and the leaf / flower modality image are scaled to 224×224 pixels using bilinear interpolation. Then, normalization is applied to both modalities, mapping pixel values ​​from [0,255] to a standardized distribution of approximately [-2,2]. The normalization parameters are: mean = [0.485, 0.456, 0.406], standard deviation std = [0.229, 0.224, 0.225]. After preprocessing, both modalities are converted to a tensor format of [3, 224, 224] dimensions.

[0198] S2, Organ Perception Feature Extraction:

[0199] The two-modal image in tensor format is input into the ResNet18 backbone network. After passing through the Conv1, Layer1, and Layer2 layers of the ResNet18 backbone network, the input feature map size becomes 128×28×28 (128 channels, 28 height, 28 width). After passing through Layer3, the input feature map size becomes 256×14×14 (256 channels, 14 height, 14 width), resulting in the input feature map X. ;

[0200] Input the input feature map X into OAE-Block:

[0201] 1) Structural reinforcement branch:

[0202] .in, This represents a convolution operator with multi-scale receptive fields. The feature map output for the structure enhancement branch;

[0203] The specific computation steps for the convolution operator of multi-scale receptive fields are as follows:

[0204] Step 1: Perform convolution operation on the input feature map X using a 3×3 convolution kernel, with padding of 1, and output feature map F1;

[0205] Step 2: Perform convolution operation on the input feature map X using a 5×5 kernel, with padding of 2, and output feature map F2;

[0206] Step 3: Perform convolution operation on the input feature map X using a 7×7 kernel, with padding of 3, and output feature map F3;

[0207] Step 4: Perform a 3×3 dilated convolution operation on the input feature map X with a dilation rate of 2 and a padding of 2, and output feature map F4;

[0208] Step 5: Concatenate feature maps F1, F2, F3, and F4 along the channel dimension to obtain the concatenated feature map F_concat;

[0209] Step 6: Perform channel integration on F_concat using 1×1 convolution to output the final structure-enhanced feature map Fs.

[0210] 2) Color and texture enhancement branch:

[0211] .in, This represents a color-texture attention operator based on a combination of channel attention and spatial attention. Enhance the feature map for color texture;

[0212] The specific calculation steps for channel attention are as follows: First, perform global average pooling on the input feature map X in the spatial dimension to compress the input feature map X into a 256×1×1 vector; Second, compress the number of channels from 256 to 16 through 1×1 convolution, and then use the ReLU activation function; Third, restore the number of channels from 16 to 256 through another 1×1 convolution, and finally use the Sigmoid function to map the values ​​to the [0,1] interval to obtain the channel weight vector;

[0213] The specific steps for calculating spatial attention are as follows: perform convolution operation on the input feature map X using a 7×7 convolution kernel to map 256 channels into 1 channel, and then use the Sigmoid function to map the values ​​to the [0,1] interval to obtain the spatial weight vector.

[0214] The channel weight vector and spatial weight vector are used to weight the original features channel-wise and spatially, respectively. That is, the original features are first multiplied by the channel weight vector, and then the result is multiplied by the spatial weight vector.

[0215] 3) Global context branch:

[0216] Global average pooling is performed on the input feature map X in the spatial dimension, thus transforming the input feature map X from... Compressed into a 256×1×1 global semantic description vector. ,in The first step is to use a global semantic description vector; the second step is to flatten the 256×1×1 global semantic description vector into a one-dimensional vector. , Step 3: Reduce the dimension from 256 to [the desired value] using a fully connected layer. Then, a nonlinear transformation is performed using the ReLU activation function. FC1 reduces the dimensions from 256 to 64. The first step is to compress the global features; the second step is to increase the dimension from 64 to 256 using a fully connected layer to obtain the global features used to guide subsequent fusion decisions. FC2 increases the dimensions from 64 to 256. This is the final global semantic vector. The fully connected layer and the ReLU activation function together constitute a nonlinear transformation layer in the form of a multilayer perceptron (MLP).

[0217] 4) Organ type embedding vector:

[0218] A learnable vector parameter with dimension 64 is initialized for both the root modality image and the leaf / flower modality image. Specifically, the organ embedding vector of the root modality is represented as follows: The organ embedding vector of the leaf-flower modality is represented as , , ,in Indicates the root modality label. This represents the leaf / flower modality label, and `Emb` is the embedding layer mapping function. When using it, the corresponding organ embedding vector is selected based on the modality type of the input image: if the input is a root modality image, then... If the input is a leaf and flower modality image, then select... .

[0219] 5) Organ-gated fusion unit:

[0220] The generation form of the gating weights is written as: ,in, Represents the global semantic vector With organ embedding vector splicing, This represents a multilayer perceptron used to generate weights. Assign weights to the corresponding structural enhancement branch, color / texture enhancement branch, and global context branch, and satisfy... The fusion feature of the three branches is represented as follows: ,in, Represented by global semantic vector The global feature map is obtained through linear mapping and expansion. Finally, the output of OAE-Block is obtained through residual connections: ,in, Organ enhancement feature map, organ enhancement feature map .

[0221] Continuing to use Layer 4 of the ResNet18 deep learning backbone network, the organ enhancement feature map is further processed. from Further extraction yields a 256×7×7 feature map; global average pooling then pools the 256×7×7 feature map into a 512-dimensional vector;

[0222] We obtain a 512-dimensional vector from 283 root modal images + a 512-dimensional vector from 283 leaf and flower modal images.

[0223] S3, Intramodal Classification Prediction

[0224] Independent classification sub-networks are constructed based on organ-enhanced feature maps to predict varieties from root modality images and leaf-flower modality images, yielding corresponding classification probability vectors. The classification sub-networks include the following structure:

[0225] (1) Fully connected classification layer: Maps the 512-dimensional vector to 12 categories (corresponding to 12 varieties of white peony), and outputs a 12-dimensional logits vector; the formula for the fully connected classification layer of the root modality classifier is as follows: The formula for the fully connected classification layer of the leaf and flower modality classifier is: ;

[0226] (2) Softmax normalization layer: converts the 12-dimensional logits vector into a 12-dimensional probability distribution vector.

[0227] Output 12-dimensional probability vectors for 283 root mode patterns and 283 leaf and flower mode patterns.

[0228] S4. Employing an uncertainty calculation method based on the entropy function { , , i=1,2,3,……,283}, calculate the entropy value for the root mode probability distribution vector, calculate the entropy value for the leaf and flower mode probability distribution vector, and obtain the uncertainty values ​​of 283 root mode images and 283 leaf and flower mode images.

[0229] Weights are generated using an inverse proportional normalization method: ;in and Let represent the fusion weights of the root mode and the leaf / flower mode, respectively, and satisfy the following: 283 sets of fusion weights were obtained. , ).

[0230] Detailed calculation example of step S4: Taking the first sample as an example: The root modality image prediction probability is The prediction probability of the leaf and flower modality image is: The entropy value is obtained by calculation. Weights are generated using inverse proportional normalization: , Repeat the above calculation for all 283 samples to obtain 283 sets of fusion weights. ).

[0231] S5. Multimodal Dynamic Fusion: Based on the generated fusion weights, the classification probability vectors of the root modality image and the leaf / flower modality image are weighted and fused. The system obtains a 12-dimensional probability vector after fusing 283 white peony samples, and outputs the white peony variety identification result based on the category corresponding to the highest probability, resulting in 283 predicted variety labels.

[0232] Comparative Example 1:

[0233] In this comparative example, ResNet18 was used as the backbone network, and only root modality images were used for variety identification. Multimodal fusion was not employed, but everything else was the same as in Example 1. The 283 white peony samples from Example 1 were input into the identification system, and 283 predicted variety labels were output.

[0234] Comparative Example 2:

[0235] In this comparative example, ResNet18 was used as the backbone network, and only leaf and flower modal images were used for variety identification. Multimodal fusion was not employed, but everything else was the same as in Example 1. The 283 white peony samples from Example 1 were input into the identification system, and 283 predicted variety labels were output.

[0236] Comparative Example 3:

[0237] In this comparative example, ResNet18 is used as the backbone network, and a simple feature concatenation and fusion method is adopted. That is, the feature vectors of the root mode and the leaf and flower mode are concatenated and then input into the classifier. The OAE-Block organ perception feature enhancement module and the DUF dynamic fusion module are not used. Everything else is the same as in Example 1. The 283 white peony samples from Example 1 are input into this recognition system, and 283 predicted variety labels are output.

[0238] Comparative Example 4:

[0239] In this comparative example, ResNet18 was used as the backbone network, and the OAE-Block organ perception feature enhancement module was adopted. However, a fixed-weight average fusion method was used, that is, the root modality image and the leaf and flower modality image each accounted for 50% of the weight. The DUF dynamic fusion module was not used, and other aspects were the same as in Example 1. The 283 white peony samples from Example 1 were input into the recognition system, and 283 predicted variety labels were output.

[0240] In this invention, the recognition accuracy = number of correctly identified samples / total number of samples. The macro-average F1 score is calculated separately for each variety, and then the arithmetic mean is taken. The F1 score calculation formula is F1 = 2 × (Precision × Recall) / (Precision + Recall), where Precision = TP / (TP + FP), Recall = TP / (TP + FN), TP represents true positives, FP represents false positives, and FN represents false negatives. The macro-average F1 score comprehensively considers both precision and recall, and can more comprehensively reflect the model's performance in multi-class classification tasks, especially when the class distribution is uneven, it is more valuable than precision. The experimental results for 283 predicted varieties are shown in the table below:

[0241] Group Recognition accuracy Macro average F1 score Example 2 95.05% 95.12% Comparative Example 1 76.68% 68.40% Comparative Example 2 68.00% 65.30% Comparative Example 3 89.40% 88.50% Comparative Example 4 91.00% 90.50%

[0242] As shown in the table above, the method of this invention (OAE-Block + DUF + conservative optimization) achieves a recognition accuracy of 95.05%, which is an improvement of 18.37% compared to the ResNet18 baseline (root modality only, without organ-sensing feature enhancement and dynamic fusion mechanism), 27.05% compared to the leaf and flower modality-only single-modality method, 5.65% compared to the feature splicing fusion method (without organ-sensing feature enhancement), and 4.05% compared to the fixed-weight average fusion method (without dynamic fusion mechanism). The method of this invention, through the DUF dynamic fusion mechanism, has significant advantages in multimodal information integration and prediction stability, and can better adapt to recognition tasks under complex acquisition conditions.

[0243] Therefore, the method of the present invention can effectively improve the accuracy and stability of white peony variety identification and has good practical application value.

Claims

1. A method for identifying white peony varieties based on multimodal organ feature fusion, characterized in that, Includes the following steps: S1. Collect and preprocess root modal images and leaf and flower modal images of Paeonia lactiflora; S2. Input the preprocessed root modality image and leaf / flower modality image into the deep learning backbone network containing the organ perception feature enhancement module. The organ perception feature enhancement module includes a structure enhancement branch, a color / texture enhancement branch, and a global context branch. The organ perception feature enhancement module generates organ-enhanced feature maps by performing conditional feature fusion by combining organ type embedding vectors and a gating fusion mechanism. S3. Based on the enhanced feature maps of organs, construct independent classification sub-networks to predict varieties in root modality images and leaf and flower modality images, and generate corresponding classification probability vectors; S4. Calculate the prediction uncertainty based on the classification probability vector; generate fusion weights based on the prediction uncertainty. S5. Based on the fusion weight, the classification probability vectors of the root modality image and the leaf and flower modality image are weighted and fused to generate a fused prediction probability vector; the white peony variety is identified based on the fused prediction probability vector.

2. The method for identifying white peony varieties based on multimodal organ feature fusion according to claim 1, characterized in that, Step S1 includes: The root modality image and the leaf / flower modality image are input into the bimodal image preprocessing module for preprocessing. The preprocessing of the root modality image includes one or more of the following: subject region extraction, size normalization, pixel value normalization, noise suppression, and conversion to tensors. The preprocessing of the leaf / flower modality image includes one or more of the following: subject region localization, size normalization, pixel value normalization, image enhancement and background weakening, and conversion to tensors. The root modality image and the leaf / flower modality image are kept consistent in size, brightness range, color space, and background complexity when input into the bimodal image preprocessing module.

3. The method for identifying white peony varieties based on multimodal organ feature fusion according to claim 1, characterized in that, In step S2, the structure enhancement branch enhances the arrangement of the main and branch roots of Paeonia lactiflora and the difference in thickness of the main and branch roots in the root modality image by using a multi-scale receptive field convolution operator, and enhances the vein direction, leaf edge contour and petal layering structure in the leaf and flower modality image; the color and texture enhancement branch enhances the color distribution and texture details of the root modality image and the leaf and flower modality image by using a color-texture attention operator based on a combination of channel attention and spatial attention; the global context branch generates a global semantic vector by global average pooling or global attention, and obtains global features through nonlinear transformation.

4. The method for identifying white peony varieties based on multimodal organ feature fusion according to claim 3, characterized in that, Step S2 further includes generating independent organ embedding vectors for the root modality image and the leaf and flower modality image respectively; the organ-gated fusion unit of the organ-aware feature enhancement module receives the outputs of the structure enhancement branch, the color and texture enhancement branch and the global context branch, and combines the organ embedding vectors and global features, and performs weighted summation of the features of each branch through a set of learnable gating weights, and then generates an organ-enhanced feature map through residual connection.

5. The method for identifying white peony varieties based on multimodal organ feature fusion according to claim 1, characterized in that, Step S4 includes: Based on the classification probability vectors of the root mode and the leaf-flower mode, the prediction uncertainty of each mode is calculated using an uncertainty calculation unit, based on the statistical function of the discreteness of the probability distribution or the inverse measure of the maximum probability value.

6. The method for identifying white peony varieties based on multimodal organ feature fusion according to claim 5, characterized in that, Step S4 also includes: Based on the prediction uncertainty, a weight generation unit is used to generate fusion weights for the root mode image and the leaf and flower mode image based on an inverse proportional function or linear normalization.

7. The method for identifying white peony varieties based on multimodal organ feature fusion according to claim 6, characterized in that, The basis for generating fusion weights includes image quality indicators; the weight generation mechanism is two-level or multi-level; the fusion method includes at least one of linear weighting and nonlinear fusion strategies.

8. The method for identifying white peony varieties based on multimodal organ feature fusion according to claim 6, characterized in that, Step S4 further includes: using a multimodal dynamic fusion module to perform weighted fusion of the classification probability vectors of the root modality image and the leaf and flower modality image; the multimodal dynamic fusion module uses multiple uncertainty indicators to jointly measure the reliability of the root modality and leaf and flower modality predictions.

9. The method for identifying white peony varieties based on multimodal organ feature fusion according to claim 1, characterized in that, White peony varieties include Anhui Changxing Dahonghua, Anhui Pubang, Shandong Fenhua, Shandong Honghua, Sichuan Double-flowered White Flower, Sichuan Double-flowered Pink Flower, Zhejiang Dahongpao, Zhejiang Gaogan, Zhejiang Chajiaogen, Zhejiang Kuanyefen, Zhejiang Wulong Tanhai, and Zhejiang Yanzhidianyue.

10. A system employing the method for identifying white peony varieties based on multimodal organ feature fusion as described in any one of claims 1-9, characterized in that, include: Image acquisition device for acquiring modal images of white peony root and leaf / flower. The data processing device includes a dual-modal image preprocessing module, a deep learning backbone network with an organ perception feature enhancement module, and a multimodal dynamic fusion module; the organ perception feature enhancement module includes a structure enhancement branch, a color and texture enhancement branch, and a global context branch; the organ perception feature enhancement module performs conditional feature fusion by combining organ type embedding vectors and a gating fusion mechanism; The storage device is used to save root modality images, leaf and flower modality images, and data generated during the deep learning recognition process, including parameters, recognition logs, and historical recognition results. The output device is used to output the recognition content, including the white peony variety name, prediction probability, and fusion weight.

Images