An image-spectrum bimodal feature fusion method for detecting impurities in leafy Chinese herbal medicine
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-15
- Publication Date
- 2026-08-11
AI Technical Summary
[0008]本发明的目的在于提供一种图像-光谱双模态特征融合的叶类中草药杂质检测方法以解决背景技术中所提到的问题
[0017]第一，本发明同时利用RGB图像的空间纹理信息和光谱反射信息，弥补单一RGB图像中草药与杂质颜色相近或纹理相似导致的检测不足，同时解决单一光谱信息中空间纹理细节不足的问题。
Smart Images
Figure CN122551082A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision and hyperspectral imaging detection, and particularly relates to a method for detecting impurities in leafy Chinese herbal medicines by image-spectral dual-modal feature fusion. Background Technology
[0002] During the harvesting, drying, and transportation of leafy Chinese medicinal herbs, impurities such as leaves, branches, soil, snail shells, plastic fragments, and cigarette butts can easily contaminate them, affecting their medicinal value and food safety. Therefore, rapid and accurate identification and filtering of impurities are of great importance in the production and processing of leafy Chinese medicinal herbs.
[0003] Currently, impurity detection in leafy Chinese herbal medicines mainly relies on manual sorting. While simple and easy to learn, this method is labor-intensive, inefficient, and prone to subjectivity, leading to poor standard consistency and failing to meet the demands of large-scale and automated production. With the development of machine vision technology, traditional RGB image processing methods and deep learning-based detection methods are increasingly being applied to tasks such as medicinal material detection and quality grading. These methods primarily utilize spatial structural information such as color, texture, and shape for judgment, achieving certain results when the background is simple and the impurities are clearly distinguishable from the medicinal materials.
[0004] However, leafy medicinal herbs are inherently complex in form, exhibiting curling, overlapping, breakage, obscuring, and significant color variations. Some impurities are similar in color and texture to leafy herbs, and are small in size with irregular boundaries, easily leading to missed or false detections. Furthermore, RGB imaging is susceptible to the effects of light intensity, shooting angle, surface shadows, and background conditions. Relying solely on RGB imaging is insufficient to fully characterize the material differences between medicinal herbs and impurities, resulting in poor model generalization ability.
[0005] Spectral imaging can simultaneously acquire spectral information of a target across multiple bands, reflecting differences in material composition, moisture content, and surface reflectivity. It has high application value in agricultural product grading, medicinal herb identification, and foreign object detection. For targets with similar colors or textures, spectral information can provide spectral discrimination features that are difficult to express with RGB imaging. However, existing hyperspectral information typically contains hundreds or even thousands of bands, resulting in high data dimensionality and a lot of redundant information. Directly using this information in deep learning models significantly increases computational and storage costs, leading to training difficulties, slow inference speeds, and an increased risk of overfitting. Existing band selection methods partially rely on human experience or statistical indicators, which may not be directly related to the actual segmentation and detection task, making it difficult to guarantee that the selected bands have optimal discrimination capabilities for target recognition.
[0006] Furthermore, existing RGB imaging and spectral fusion methods mostly employ simple channel stitching or shallow feature fusion, failing to fully consider the complementary relationship between RGB spatial texture features and spectral material features, and lacking multi-scale fusion mechanisms for instance-level impurity segmentation. For impurities with irregular shapes and arbitrary orientations, simply outputting the category or horizontal bounding box often fails to accurately represent their true position and orientation, which is detrimental to subsequent automatic sorting and robotic gripping.
[0007] Therefore, there is an urgent need for a detection method that can fully integrate RGB image information and spectral information to achieve fine segmentation and orientation localization of impurities while reducing spectral redundancy. Summary of the Invention
[0008] The purpose of this invention is to provide a method for detecting impurities in leafy Chinese herbal medicines by image-spectral dual-modal feature fusion to solve the problems mentioned in the background art.
[0009] To achieve the above objectives, the present invention adopts the following technical solution:
[0010] A method for detecting impurities in leafy Chinese herbal medicines based on image-spectral dual-modal feature fusion includes the following steps:
[0011] S1. Obtain RGB images, spectral information, and image annotations for band selection training of leafy Chinese medicinal herbs and impurities; generate a binary classification segmentation mask based on the image annotations to construct positive and negative samples;
[0012] S2. Read the 304 spectral bands corresponding to the positive and negative samples from the spectral information, and perform standardization and cropping on the 304 spectral bands.
[0013] S3. Input the processed 304 spectral bands into the TaskGate-k16 band selection model, and select 16 spectral bands from the 304 spectral bands using the TaskGate-k16 model.
[0014] S4. Based on the 16 spectral band indices, the original spectral information is sliced to obtain 16-channel spectral information; the 3-channel RGB image is spatially aligned and channel-fused with the 16-channel spectral information to obtain 19-channel fused data.
[0015] S5. Input the fused data into the RGB-spectral dual-branch fusion instance segmentation network. The RGB-spectral dual-branch fusion instance segmentation network includes an image spectral branch (RGB HIS Stem), an RGB dominant backbone network, spectral branches, a gated residual fusion module, a multi-scale feature fusion module, and a segmentation detection head. The segmentation detection head outputs the category, confidence level, and instance mask of the impurity instance. Based on the instance mask, a minimum bounding rectangle rotation box is generated, and the visualization results are output.
[0016] Compared with existing technologies, this invention provides a method for detecting impurities in leafy Chinese herbal medicines based on image-spectral dual-modal feature fusion, which has the following beneficial effects:
[0017] First, the present invention utilizes both the spatial texture information and spectral reflectance information of RGB images to compensate for the insufficient detection caused by the similarity in color or texture between herbs and impurities in a single RGB image, while also solving the problem of insufficient spatial texture details in single spectral information.
[0018] Second, this invention proposes the TaskGate-k16 band selection method, which uses a segmentation mask to construct foreground positive samples, boundary ring negative samples, and far background negative samples, making the band selection process related to the impurity instance segmentation task. It selects 16 original spectral bands from 304 original spectral bands, reducing spectral redundancy channels and lowering the cost of data storage, model training, and inference computation.
[0019] Third, this invention sets up an RGB HIS Stem to achieve residual fusion of RGB spatial features and spectral material features in the shallow layer of the network, so that spectral information can participate in target representation in the early feature encoding stage.
[0020] Fourth, this invention sets up a spectral branch to independently retain multi-scale hyperspectral discrimination information, thus avoiding the weakening of spectral information during the RGB dominant feature extraction process.
[0021] Fifth, this invention sets up a gated residual fusion module to adaptively control the injection ratio of spectral information, enabling the model to dynamically fuse dual-modal information based on differences in target material and background features. Attached Figure Description
[0022] Figure 1 This is a flowchart of the overall process for a method of detecting impurities in leafy Chinese herbal medicines based on image-spectral dual-modal feature fusion proposed in this invention.
[0023] Figure 2 This is a schematic diagram of the RGB HIS Stem and early RGB-spectral residual fusion module proposed in this invention.
[0024] Figure 3 This is a schematic diagram of the spectral independent multi-scale feature extraction branch proposed in this invention.
[0025] Figure 4 This is a schematic diagram of the gated residual fusion module structure proposed in this invention.
[0026] Figure 5 This is a schematic diagram of the 16-channel hyperspectral band selection proposed in this invention. Detailed Implementation
[0027] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.
[0028] Example 1:
[0029] The following experiments were conducted using a self-built dataset, the information of which is shown in Table 1:
[0030] Table 1. Dataset-related information
[0031] Dataset partitioning picture background Example Training set (train) 2528 478 4809 Validation set (val) 316 60 590 Test set (test) 316 60 594
[0032] Please see Figure 1 As shown in the figure, this embodiment proposes a method for detecting impurities in leafy Chinese herbal medicines by image-spectral dual-modal feature fusion, which specifically includes the following steps:
[0033] S1. Obtain RGB images, spectral information, and image annotations for band selection training of leafy Chinese medicinal herbs and impurities; generate a binary classification segmentation mask based on the image annotations to construct positive and negative samples. Specifically, this includes the following:
[0034] The formula for black-and-white correction of spectral information is as follows:
[0035]
[0036] Where I represents the original spectral information and D represents the dark-field image. The image represents the whiteboard image, and R represents the corrected reflectance information.
[0037] All impurity polygon annotations in the same image are merged to form a joint foreground mask; pixels within the joint foreground mask are identified as positive foreground samples.
[0038] The joint foreground mask is dilated, and the dilated ring region around the mask is defined as the annular negative sample region; the background region far from the joint foreground mask is defined as the far background negative sample region; the final negative sample includes the annular negative sample region and the far background negative sample region.
[0039] S2. Read 304 spectral bands corresponding to positive and negative samples from the spectral information, and perform standardization and cropping on these 304 spectral bands. Specifically, this includes the following:
[0040] The spectral bands are standardized based on their mean and standard deviation, using the following formula:
[0041]
[0042] in, Represents 304-dimensional spectral information. This represents the mean value across all bands. This represents the standard deviation of each band.
[0043] The standardized pixel values are cropped to a preset range, which is set to [-6, 6].
[0044] S3. Input the processed 304 spectral bands into the TaskGate-k16 band selection model, and select 16 spectral bands from the 304 spectral bands using the TaskGate-k16 model. Specifically, this includes the following:
[0045] The TaskGate-k16 band selection model includes a Top-K module and a binary classification surrogate module. The Top-K module assigns a learnable logit to each original spectral band and calculates the band importance score according to the following formula:
[0046]
[0047] in, This represents the learnable parameters corresponding to the 304 original spectral bands. Indicates temperature parameter, This represents the soft fraction corresponding to each band.
[0048] Top-K selection is performed based on the band importance score, where k=16, resulting in hard selection masks hard_mask for 16 bands.
[0049] Constructing a band-selective mask capable of backpropagation using a through-pass estimation method:
[0050] mask = hard_mask - scores.detach() + scores
[0051] The input spectral information is masked by selecting a mask based on the band, resulting in:
[0052] x_selected = x mask
[0053] Where x_selected represents the spectral information after band selection masking.
[0054] The binary classification proxy discrimination module is a multilayer perceptron, used to output the discrimination result of whether a pixel belongs to the impurity foreground or background after the spectral information is masked by band selection.
[0055] The training loss of the binary classification agent discrimination module is BCE With Logits Loss, and the weight of positive samples can be set according to the ratio of positive to negative samples.
[0056] During the validation phase, Precision, Recall, and F1 are calculated at a fixed threshold, and the optimal threshold that maximizes F1 is searched within the predicted probability range. The best F1 on the validation set is used as the model selection criterion, and the optimal TaskGate-k16 band selection model is selected, and the corresponding 16 original spectral bands are derived: 1, 4, 33, 49, 109, 122, 134, 139, 203, 213, 230, 268, 287, 293, 297, 303, as shown in Table 2.
[0057] Table 2.16 Original Spectral Band Information
[0058] Serial Number Cache channel number Original band index Wavelength (nm) 1 0 1 396.35 2 1 4 402.60 3 2 33 462.80 4 3 49 495.81 5 4 109 618.36 6 5 122 644.67 7 6 134 668.89 8 7 139 678.97 9 8 203 806.96 10 9 213 826.81 11 10 230 860.47 12 11 268 935.37 13 12 287 972.65 14 13 293 984.40 15 14 297 992.23 16 15 303 1003.96
[0059] S4. Based on the 16 spectral band indices, the original spectral information is sliced to obtain 16-channel spectral information; the RGB image and the 16-channel spectral information are spatially aligned and fused to obtain fused data. For example... Figure 2 As shown, it specifically includes the following:
[0060] Sixteen original spectral bands were extracted from 304 original spectral bands based on an index. The RGB image 3-channel (R (red), G (green), B (blue)) and 16-channel spectral information corresponding to the same sample were read. The RGB image 3-channel and spectral information 16-channel were combined in the channel dimension to obtain 19-channel fused input data.
[0061] S5. Input the fused data into the RGB-spectral dual-branch fusion instance segmentation network. The RGB-spectral dual-branch fusion instance segmentation network includes an image spectral branch (RGB HIS Stem), an RGB dominant backbone network, spectral branches, a gated residual fusion module, a multi-scale feature fusion module, and a segmentation detection head. The segmentation detection head outputs the category, confidence level, and instance mask of the impurity instance. Based on the instance mask, a minimum bounding box with rotation is generated, and the visualization results are output. Figure 3 , 4 As shown in Figure 5, it specifically includes the following:
[0062] The image spectral branch (RGB HIS Stem) includes the RGB branch and the spectral projection branch, where:
[0063] The RGB branch is used to extract spatial texture features of RGB images; the spectral projection branch includes a 1×1 spectral projection layer and a convolutional layer, which are used to map 16-channel spectral information to a feature space that is compatible with RGB features.
[0064] The RGB HIS Stem fuses the initial features as follows:
[0065]
[0066] in, Indicates the output characteristics of the RGB branch. Indicates the output characteristics of the spectral branch. This represents learnable fusion weights.
[0067] The RGB-dominant backbone network is used to extract RGB spatial texture features, the spectral branches are used to extract spectral multi-scale material features, and the gated residual fusion module fuses RGB P4 features and hyperspectral P4 features at the P4 scale. The fusion formula is as follows:
[0068]
[0069] in, This represents the fused P4 features. This represents the characteristics of the RGB main branch output at the P4 scale. This indicates the characteristics of the spectral branch output at the P4 scale. This represents the 1×1 projection mapping used for channel alignment. α is a trainable global scalar gating parameter, which, after being normalized by the Sigmoid function, becomes the adaptive scaling factor for the spectral residual branch. Initially, it is negative to make the hyperspectral residual signal weaker in the early stages of training.
[0070] The multi-scale feature fusion module includes the SPPF module, the C2PSA module, and the FPN / PAN Neck module. The fused features are processed by the SPPF and C2PSA modules and then input into the FPN / PAN Neck module; the FPN / PAN Neck module outputs features at three scales: P3, P4, and P5.
[0071] The segmentation detection head outputs the category, confidence level, horizontal bounding box, and instance mask of impurity instances based on features at three scales: P3, P4, and P5. It calculates the minimum bounding rectangle based on the instance mask contour to obtain the bounding box. It outputs the center point, width, height, and rotation angle of the bounding box and finally outputs the visualization results.
[0072] The experimental results for the validation set and the test set are shown in Table 3 and Table 4, respectively.
[0073] Table 3. Val set experimental results
[0074] index Box Mask P 0.973 0.972 R 0.954 0.953 mAP50 0.985 0.980 mAP50-95 0.847 0.730
[0075] Table 4. Experimental Results of the Test Set
[0076] index Box Mask P 0.984 0.984 R 0.953 0.953 mAP50 0.982 0.978 mAP50-95 0.865 0.744
[0077] The indicators in the table mean the following:
[0078] P: Precision. Indicates the percentage of targets predicted as impurities by the model that are actually impurities. A higher value indicates fewer false positives.
[0079] R: Recall. It indicates how many actual impurities are detected by the model. A higher value indicates fewer missed detections.
[0080] mAP50: The average accuracy calculated under the condition of IoU=0.50, which mainly reflects whether the model can find the target relatively accurately.
[0081] mAP50-95: The average precision calculated at multiple thresholds from IoU=0.50 to 0.95, which is more stringent than mAP50 and better reflects the fineness of the bounding box or segmentation mask.
[0082] Box: The object detection bounding box metric, used to evaluate the accuracy of the impurity location bounding boxes.
[0083] Mask: Instance segmentation mask metric, used to evaluate the accuracy of impurity contour regions.
[0084] The impurity binary classification instance segmentation model trained on 3160 mixed samples (training:validation:test = 8:1:1) achieved superior performance on both the validation and test sets, indicating that the model has strong detection and segmentation capabilities for impurity targets. Test set results show that the model's Box and Mask precision both reach 0.984, and the recall rate is 0.953, demonstrating that the model can effectively identify most impurity targets in real-world test samples with a low false positive rate.
[0085] In terms of localization performance, the Box mAP50 score reached 0.982 and the Box mAP50-95 score reached 0.865, indicating that the model accurately predicts the location of impurity targets and maintains good detection stability even under stricter IoU threshold conditions. Regarding segmentation performance, the Mask mAP50 score was 0.978 and the Mask mAP50-95 score was 0.744, indicating that the model can accurately extract the contours of impurity regions. However, under high IoU thresholds, there is still room for improvement in the segmentation boundary accuracy.
[0086] The results on the validation set and the test set are generally similar, and the Mask mAP50-95 on the test set is slightly higher than that on the validation set, indicating that the model does not show obvious overfitting and has good generalization ability.
Claims
1. A method for detecting impurities in leafy Chinese herbal medicines using image-spectral dual-modal feature fusion, characterized in that, Includes the following steps: S1. Obtain RGB images, spectral information, and image annotations for band selection training of leafy Chinese medicinal herbs and impurities; generate a binary classification segmentation mask based on the image annotations to construct positive and negative samples; S2. Read the 304 spectral bands corresponding to the positive and negative samples from the spectral information, and perform standardization and cropping on the 304 spectral bands. S3. Input the processed 304 spectral bands into the TaskGate-k16 band selection model, and select 16 spectral bands from the 304 spectral bands using the TaskGate-k16 model. S4. Based on the 16 spectral band indices, the original spectral information is sliced to obtain 16-channel spectral information; the 3-channel RGB image is spatially aligned and channel-fused with the 16-channel spectral information to obtain 19-channel fused data. S5. Input the fused data into the RGB-spectral dual-branch fusion instance segmentation network. The RGB-spectral dual-branch fusion instance segmentation network includes the image spectral branch RGB HIS Stem, the RGB dominant backbone network, the spectral branch, the gated residual fusion module, the multi-scale feature fusion module, and the segmentation detection head. The segmentation detection head outputs the category, confidence level, and instance mask of the impurity instance. Based on the instance mask, the minimum bounding rectangle of the rotation is generated, and the visualization results are output.
2. The method for detecting impurities in leafy Chinese herbal medicines using image-spectral dual-modal feature fusion according to claim 1, characterized in that, S1 specifically includes the following: The formula for black-and-white correction of spectral information is as follows: Where I represents the original spectral information and D represents the dark-field image. The image represents a whiteboard image, and R represents the corrected reflectance information. All impurity polygon annotations in the same image are merged to form a joint foreground mask; pixels within the joint foreground mask are identified as positive foreground samples. The joint foreground mask is dilated, and the dilated ring region around the mask is defined as the annular negative sample region; the background region far from the joint foreground mask is defined as the far background negative sample region; the annular negative sample region and the far background negative sample region together constitute the negative sample.
3. The method for detecting impurities in leafy Chinese herbal medicines using image-spectral dual-modal feature fusion according to claim 2, characterized in that, S2 specifically includes the following: The spectral bands are standardized based on their mean and standard deviation, using the following formula: in, Represents 304-dimensional spectral information. This represents the mean value across all bands. Indicates the standard deviation of each band; The standardized pixel values are cropped to a preset range.
4. The image-spectra dual-modal feature fusion method for detecting impurities in leafy Chinese herbal medicine according to claim 3, characterized in that, S3 specifically includes the following: The TaskGate-k16 band selection model includes a Top-K module and a binary classification surrogate discriminant module. The Top-K module assigns a learnable logit to each original spectral band and calculates the band importance score according to the following formula: wherein, denote 304 learnable parameters corresponding to the original spectral bands, denote temperature parameters, denote soft scores corresponding to each band; Top-K selection is performed based on the band importance score, where k=16, resulting in hard selection masks hard_mask for 16 bands; Constructing a band-selective mask capable of backpropagation using a through-pass estimation method: mask = hard_mask - scores.detach() + scores The input spectral information is masked by selecting a mask based on the band, resulting in: x_selected = x mask Where x_selected represents the spectral information after band selection masking; The binary classification agent discrimination module is a multilayer perceptron, which is used to output the discrimination result of whether the pixel belongs to the impurity foreground or background after the spectral information is masked by band selection. The training loss of the binary classification agent discrimination module is BCE With Logits Loss, and the weight of positive samples can be set according to the ratio of positive to negative samples. During the validation phase, Precision, Recall, and F1 are calculated at a fixed threshold, and the optimal threshold that maximizes F1 is searched within the predicted probability range. The best F1 on the validation set is used as the model selection criterion to select the optimal TaskGate-k16 band selection model and derive the corresponding 16 original spectral bands: 1, 4, 33, 49, 109, 122, 134, 139, 203, 213, 230, 268, 287, 293, 297, and 303.
5. The image-spectra dual-modal feature fusion method for detecting impurities in leafy Chinese herbal medicine according to claim 4, characterized in that, S4 specifically includes the following: Sixteen original spectral bands were extracted from 304 original spectral bands based on an index. Read the RGB image and 16-channel spectral information corresponding to the same sample; combine the 3 channels of the RGB image and the 16 channels of the spectral information in the channel dimension to obtain 19-channel fused input data.
6. The method for detecting impurities in leafy Chinese herbal medicines using image-spectral dual-modal feature fusion according to claim 5, characterized in that, S5 specifically includes the following: The RGB HIS Stem includes an RGB branch and a spectral projection branch, where: The RGB branch is used to extract spatial texture features of RGB images; the spectral projection branch includes a 1×1 spectral projection layer and a convolutional layer, which are used to map 16-channel spectral information to a feature space that matches the RGB features. The RGB HIS Stem fuses the initial features as follows: in, Indicates the output characteristics of the RGB branch. Indicates the output characteristics of the spectral branch. This represents learnable fusion weights; The RGB-dominant backbone network is used to extract RGB spatial texture features, the spectral branches are used to extract spectral multi-scale material features, and the gated residual fusion module fuses RGB P4 features and hyperspectral P4 features at the P4 scale. The fusion formula is as follows: in, This represents the fused P4 features. This represents the characteristics of the RGB main branch output at the P4 scale. This indicates the characteristics of the spectral branch output at the P4 scale. This represents a 1×1 projection mapping used for channel alignment, where α is a trainable global scalar gate parameter, which, after being normalized by the Sigmoid function, becomes the adaptive scaling factor for the spectral residual branch. The multi-scale feature fusion module includes the SPPF module, the C2PSA module, and the FPN / PAN Neck module. The fused features are processed by the SPPF and C2PSA modules and then input into the FPN / PAN Neck module. The FPN / PAN Neck module outputs features at three scales: P3, P4, and P5. The segmentation detection head outputs the category, confidence level, horizontal bounding box, and instance mask of impurity instances based on features at three scales: P3, P4, and P5. It calculates the minimum bounding rectangle based on the instance mask contour to obtain the bounding box. It outputs the center point, width, height, and rotation angle of the bounding box.