Method for detecting authenticity of pericarpium citri reticulatae from Sichuan of China based on multi-modal fusion

Abstract

The invention discloses a multimodal fusion-based true and false detection method for pericarpium citri reticulatae. The method comprises the following steps: firstly, synchronously acquiring near infrared spectrum data, high-resolution image data and Raman chemical fingerprint data of pericarpium citri reticulatae; performing preprocessing operations such as median filtering, CLAHE image enhancement and DTW peak value alignment on the acquired data; then constructing a cross-modal feature fusion network, extracting corresponding features through each branch, and performing dynamic weighted fusion by a gating fusion layer to obtain a fusion feature vector; and finally, inputting the fused feature vector into a support vector machine (SVM), calibrating the output authenticity probability and 95% confidence interval by adopting a PlattScaling method, and performing authenticity judgment on the pericarpium citri reticulatae through a dynamic threshold decision engine. According to the invention, the method combines a deep learning model and a dynamic decision-making mechanism, achieves the high-precision and non-destructive authenticity identification of the traditional Chinese medicinal materials, and is suitable for the automatic sorting and anti-counterfeiting traceability of the production, circulation and market supervision links of the traditional Chinese medicinal materials.

Description

Technical Field

[0001] This invention relates to the field of traditional Chinese medicine quality testing technology, and in particular to a method for detecting the authenticity of Sichuan tangerine peel based on multimodal fusion. Background Technology

[0002] As an important component of traditional Chinese medicine, the quality of medicinal herbs directly affects clinical efficacy and medication safety. Sichuan tangerine peel (Chenpi), a commonly used medicinal herb, possesses properties such as regulating qi and strengthening the spleen, drying dampness and resolving phlegm, and has wide applications in medicine and health care. However, due to high market demand and price, a large number of counterfeit Sichuan tangerine peel products have appeared on the market. These counterfeit products not only have lower efficacy but may even harm human health. Therefore, accurate and rapid identification of genuine Sichuan tangerine peel is of significant practical importance.

[0003] Existing technologies for detecting the authenticity of Sichuan tangerine peel have many limitations, mainly in the following aspects:

[0004] Single-modal detection has significant limitations. Near-infrared spectroscopy (NIRS) can only reflect the chemical composition information of Sichuan tangerine peel and cannot identify its physical morphological differences. Some counterfeit products imitate the appearance of Sichuan tangerine peel through dyeing, adulteration, etc., which makes it difficult for NIRS to accurately identify them. Image detection methods are easily affected by factors such as lighting conditions and shooting angles, and have a weak ability to capture the microscopic texture features of the surface of Sichuan tangerine peel, resulting in low detection accuracy. Although chemical fingerprinting (such as HPLC) can reflect the chemical composition of Sichuan tangerine peel relatively accurately, it requires destructive sampling and cannot achieve real-time detection, which is not conducive to rapid screening in production, distribution and other stages.

[0005] Insufficient model generalization is also a major problem with existing technologies: traditional detection methods such as PLS regression have not solved the problem of cross-modal feature alignment for samples from multiple origins. When the samples being tested come from different origins, the test error fluctuates significantly. Because Sichuan tangerine peel from different origins differs in terms of growing environment and cultivation methods, its characteristic information will also differ. Traditional methods are difficult to adapt to these differences, resulting in poor generalization ability of the detection model, which cannot meet the actual detection needs.

[0006] Therefore, there is an urgent need for a multimodal data fusion method that combines deep feature extraction and dynamic decision-making mechanisms to break through the bottleneck of a single data source and improve the accuracy and robustness of the detection of genuine and counterfeit Sichuan tangerine peel, so as to meet the actual needs of the production, circulation and market supervision of Chinese medicinal materials. Summary of the Invention

[0007] The purpose of this invention is to provide a method for detecting the authenticity of Sichuan tangerine peel based on multimodal fusion. This method aims to solve the problems of limited single-modal detection and insufficient model generalization in existing Sichuan tangerine peel authenticity detection technologies. By fusing multimodal data such as near-infrared spectroscopy, high-resolution images, and Raman chemical fingerprints, and combining cross-modal feature fusion networks and dynamic threshold decision engines, this invention achieves high-precision, non-destructive detection of the authenticity of Sichuan tangerine peel, and improves the generalization ability and anti-interference ability of the detection model.

[0008] To achieve the above objectives, this invention provides a method for detecting the authenticity of Sichuan tangerine peel based on multimodal fusion, comprising the following steps:

[0009] S1. Simultaneously acquire multimodal data of Sichuan tangerine peel, including near-infrared spectral data, high-resolution image data, and Raman chemical fingerprint data;

[0010] S2. Preprocess the acquired multimodal data using median filtering, CLAHE image enhancement, and DTW peaking.

[0011] S3. Construct a cross-modal feature fusion network CMF-Net to extract and dynamically weight and fuse multimodal features to obtain the fused feature vector;

[0012] S4. Based on Support Vector Machine (SVM) and Plattscaling, output the true and false probabilities and confidence intervals of the Sichuan tangerine peel samples, and use a dynamic threshold decision engine to determine their authenticity.

[0013] Preferably, in S1, multimodal data of Sichuan tangerine peel are collected from the same detection area of ​​the same sample under the same detection environment. Specifically:

[0014] Near-infrared spectral data of Sichuan tangerine peel were collected using a near-infrared spectrometer with a wavelength range of 1000nm-2500nm. The integration time was set to 50ms, the incident angle was controlled at 45°±5°, and the light source intensity was greater than or equal to 90%.

[0015] High-definition cameras with a resolution higher than 3840×2160 were used to collect surface detail image data of Sichuan tangerine peel, including its surface texture and color characteristics;

[0016] Raman fingerprint data were obtained by combining high performance liquid chromatography (HPLC) with Raman spectroscopy to reflect the chemical composition and relative content of tangerine peel.

[0017] Preferably, in S2, the acquired near-infrared spectral data is first subjected to median filtering to remove noise interference, and the median filtering window is set to 9.

[0018] Then perform regularization, the regularization formula is:

[0019]

[0020] Where Z represents the regularized spectral data, X represents the original spectral data, μ represents the spectral mean, and σ represents the spectral standard deviation.

[0021] Preferably, in S2, ambient light compensation is first performed on the collected surface detail image data of Sichuan tangerine peel. The ambient light compensation uses the gray-world algorithm, and the expression is as follows:

[0022]

[0023] Where μ' is the mean gray value of the current image, H and W are the height and width of the image, I(i,j) is the original pixel value of the pixel with coordinates (i,j) in the image, and i and j represent the row index and column index of the image in the spatial dimension, respectively;

[0024] definition: Where, μ l Ideal grayscale value;

[0025] Then the compensated pixel value I b (i,j) is:

[0026]

[0027] The compensated data will be stored in PNG format;

[0028] Then, the contrast-limited adaptive histogram equalization (CLAHE) algorithm is used for image enhancement. The steps of the CLAHE algorithm are as follows:

[0029] Divide the image into multiple non-overlapping small blocks;

[0030] Perform histogram equalization on each small block separately;

[0031] The processing results of each small block are stitched together using bilinear interpolation to obtain the final enhanced image, thereby improving the image's clarity and detail.

[0032] Preferably, in S2, the acquired Raman chemical fingerprint data undergoes baseline correction and dynamic time warping (DTW) peak alignment processing.

[0033] Baseline correction is based on an adaptive smoothing algorithm to remove baseline drift interference in the spectrum;

[0034] The dynamic time warping expression is as follows:

[0035] D T (i',j')=d(i',j')+min{D T (i'-1,j'),D T(i',j'-1),D T (i'-1,j'-1)};

[0036] In the formula, D T (i',j') represents the dynamic time-warped cumulative distance obtained when matching the first i' feature points of the first fingerprint sequence with the first j' feature points of the second fingerprint sequence; d(i',j') refers to the local distance between the i'th feature point of the first fingerprint sequence and the j'th feature point of the second fingerprint sequence; D T (i'-1,j') represents the cumulative distance when retaining the j'-th feature point of the second fingerprint sequence and backtracking to the i'-1-th feature point of the first fingerprint sequence; D T (i',j'-1) represents the cumulative distance when backtracking from the i'-th feature point of the first fingerprint sequence to the j'-1-th feature point of the second fingerprint sequence, while retaining the i'-th feature point of the first fingerprint sequence; D T (i'-1,j'-1) represents the cumulative distance when the current i'th and j'th feature points are matched after the first i'-1 feature points of the first fingerprint sequence and the first j'-1 feature points of the second fingerprint sequence have been matched.

[0037] Preferably, in S3, the cross-modal feature fusion network CMF-Net includes a spectral branch, an image branch, a chemical branch, and a gated fusion layer GFU;

[0038] The spectral branch employs an improved ResNet-18 network with an added SE attention module. The ResNet-18 network addresses the vanishing gradient problem during deep neural network training through residual connections, and its output expression is:

[0039] y = F(x, {W}) + x;

[0040] In the formula, y is the output, x is the input, F(x,W) is the residual function, and W is the network parameter;

[0041] The SE attention module weights the channel features, specifically through the following process:

[0042] First, perform global average pooling:

[0043]

[0044] Among them, z c For the global pooling result of channel c, u c (i",j") represents the feature value located at the i"th row and j"th column position in the feature map of the cth channel;

[0045] Then perform channel weight calculation:

[0046] sc =σ'(W2δ(W1z) c ));

[0047] Where, σ ' δ is the Sigmoid function, δ is the ReLU function, and W1 and W2 are the weights of the fully connected layer.

[0048] Finally, feature weighting is performed:

[0049]

[0050] in, This represents the weighted feature of the c-th channel.

[0051] Preferably, in S3, the image branch uses the ViT-B16 model combined with the multi-scale feature pyramid (FPN) to extract local texture and global morphological features. Specifically:

[0052] The ViT-B16 model segments the image into P×P small regions. Each region is mapped to a vector through a linear embedding layer. After adding positional encoding, the vector is input into a multi-head self-attention Transformer encoder, which consists of a multi-head attention layer (MSA) and a multilayer perceptron (MLP). The output of the g-th layer is calculated as follows:

[0053] Z' g =MSA(LN(Z) g-1 ))+Z g-1 ;

[0054] Z g =MLP(LN(Z') g ))+Z' g ;

[0055] In the formula, Z g Z' represents the final output of the g-th layer. g Z represents the intermediate result after calculation using the multi-head self-attention mechanism MSA, added to the output residual of the (g-1)th layer; g-1 Represents the final output of the (g-1)th layer; MSA(·) denotes the multi-head self-attention mechanism; LN(·) is the layer normalization operation; MLP(·) is the multilayer perceptron.

[0056] Then, feature maps F1, F2, ..., F of different layers of the ViT-B16 model are obtained through the multi-scale feature pyramid FPN. m The resolutions corresponding to each feature map are r1, r2, ..., r m Features at different scales are fused through bilinear interpolation upsampling and lateral connectivity.

[0057] Preferably, in S3, the chemical branch uses a bidirectional long short-term memory network (Bi-LSTM) to parse the temporal information in the Raman chemical fingerprint data. The Bi-LSTM network consists of a forward LSTM and a backward LSTM.

[0058] Preferably, in S3, the gated fusion layer GFU dynamically weights and fuses multimodal features using the sigmoid function, as shown in the formula:

[0059] ω o =σ'(W o ·[f spec ,f img ,f chem ]+B);

[0060] In the formula, ω o Let W be the weight of the o-th mode, where o = 1, 2, 3 correspond to the spectral, image, and chemical modes, respectively. σ' is the Sigmoid activation function. o Let f be the modal weight matrix. spec ,f img ,f chem [] represents the concatenated multimodal feature vector; f spec For the spectral branch output characteristics, f img For image branch output features, f chem The output features are for the chemical branch, and b is the bias vector;

[0061] The formula for calculating the fused feature vector is:

[0062]

[0063] Among them, f fusion For the final fused feature vector, f o is the original feature vector of the o-th mode.

[0064] Preferably, in S4, the feature vector fused from S3 is input into a support vector machine (SVM) for classification. The decision output of the SVM is calibrated using the Plattscaling method to obtain the true and false probabilities of the Sichuan tangerine peel samples. The confidence interval is calculated and output based on the normal distribution assumption of the true and false probabilities.

[0065] The classification threshold is adaptively adjusted by a dynamic threshold decision engine, and the true or false judgment is made based on the true or false probability and confidence interval.

[0066] Therefore, the present invention employs the above-mentioned method for detecting the authenticity of Sichuan tangerine peel based on multimodal fusion, and the beneficial effects are as follows:

[0067] (1) This invention, through the construction of a multimodal data fusion and cross-modal feature fusion network CMF-Net, achieves an accuracy rate of ≥99.2% for identifying genuine and counterfeit Sichuan tangerine peel, which is 12.5% ​​higher than that of a single spectral model. The false detection rate is ≤0.5%, which can effectively identify physical morphological counterfeits and chemical component adulteration, thus solving the limitations of single-modal detection.

[0068] (2) This invention combines a dynamic threshold decision engine with a multi-origin mixed training strategy. The model can adaptively adjust the classification threshold to adapt to the differences in different origins such as Sichuan, Fujian, and Vietnam. Through preprocessing such as median filtering and CLAHE image enhancement, as well as dynamic weighting of the gated fusion layer, it can effectively resist interference from lighting, angle, baseline drift, etc., and ensure detection stability in complex environments.

[0069] (3) This invention can simultaneously acquire near-infrared spectroscopy, images and Raman chemical fingerprint data without destructive sampling, realizing non-contact detection; the online testing throughput reaches 220 samples / minute, the power consumption is ≤25W, it supports industrial production line deployment, and is suitable for automated sorting and anti-counterfeiting traceability in the production, circulation and market supervision of Chinese medicinal materials, taking into account both detection efficiency and practical value.

[0070] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0071] Figure 1 This is an overall flowchart of the method for detecting the authenticity of Sichuan tangerine peel based on multimodal fusion, according to an embodiment of the present invention.

[0072] Figure 2 This is a curve comparing the accuracy of the multimodal fusion-based method for detecting the authenticity of Sichuan tangerine peel with that of a single spectral model. Detailed Implementation

[0073] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0074] Unless otherwise defined, the technical or scientific terms used in this invention shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains.

[0075] like Figure 1 As shown, the method for detecting the authenticity of Sichuan tangerine peel based on multimodal fusion includes the following steps:

[0076] S1. Simultaneously acquire multimodal data of Sichuan tangerine peel, including near-infrared spectral data, high-resolution image data, and Raman chemical fingerprint data.

[0077] This invention collects multimodal data of Sichuan tangerine peel from the same sample in the same detection area under the same detection environment, with a detection temperature of 20-25℃ and a relative humidity of 40-60%. Specifically:

[0078] Near-infrared spectral data of Sichuan tangerine peel were collected using a near-infrared spectrometer with a wavelength range of 1000nm-2500nm. The integration time was set to 50ms to ensure stable spectral signals were acquired. The incident angle was controlled at 45°±5° to ensure uniform illumination of the tangerine peel sample and reduce measurement errors caused by angle deviation. The light source intensity was greater than or equal to 90% to provide sufficient illumination and ensure the accuracy of spectral acquisition.

[0079] High-resolution cameras with a resolution higher than 3840×2160 were used to acquire surface detail image data of Sichuan tangerine peel, including its surface texture and color characteristics. During the shooting, the tangerine peel was placed flat on a white background, and the camera lens was kept perpendicular to the sample surface at a distance of 50cm. Five images of each sample were taken from different angles to provide a high-quality data foundation for subsequent image enhancement and feature extraction.

[0080] First, high performance liquid chromatography (HPLC) was used: a C18 column (250 mm in length, 4.6 mm in inner diameter, and 5 μm in particle size) was selected, with methanol-0.1% phosphoric acid aqueous solution (40:60) as the mobile phase, a flow rate of 1.0 mL / min, a detection wavelength of 283 nm, a column temperature of 30 °C, and an injection volume of 10 μL to obtain basic data on the chemical components of dried tangerine peel.

[0081] Chemical fingerprints were then acquired using Raman spectroscopy: a 532 nm laser was used as the excitation source, with the laser power controlled between 50-100 mW to avoid thermal damage to the sample; the spectral acquisition range was 400-2000 cm⁻¹. -1 (Covering the characteristic peak range of hesperidin and other characteristic components), resolution 4cm -1 The integration time is 2-5 seconds, and three scans are performed to reduce random noise. Backscatter mode is used for acquisition, coupled with a -70℃ cooled CCD detector to reduce dark current interference. By combining these two methods, the chemical composition and relative content of Sichuan tangerine peel are comprehensively reflected.

[0082] S2. The acquired multimodal data is preprocessed using median filtering, CLAHE image enhancement, and DTW peaking.

[0083] First, the acquired near-infrared spectral data is first processed by median filtering to remove noise interference. The median filtering window is set to 9. By sorting the data in the window and taking the median value, the spectral lines are effectively smoothed and the detailed features of the spectrum are preserved.

[0084] Then perform regularization, the regularization formula is:

[0085]

[0086] Where Z represents the regularized spectral data, X represents the original spectral data, μ = 1250 represents the spectral mean, and σ = 280 represents the spectral standard deviation. Regularization maps the spectral data to the same scale, which facilitates subsequent model training and feature fusion.

[0087] Second, ambient light compensation is performed on the collected surface detail image data of Sichuan tangerine peel. The ambient light compensation uses the gray-world algorithm, and the expression is as follows:

[0088]

[0089] Where, μ ' H is the average gray level of the current image, H and W are the height and width of the image, and I(i,j) is the original pixel value of the pixel with coordinates (i,j) in the image, where i and j represent the row index and column index of the image in the spatial dimension, respectively.

[0090] definition:

[0091]

[0092] Where, μ l For ideal grayscale values, typically μ l =128;

[0093] Then the compensated pixel value I b (i,j) is:

[0094]

[0095] Ambient light compensation can eliminate the impact of different lighting conditions on image quality, making the grayscale distribution of the image more uniform. The compensated data is stored in PNG format to preserve high-quality image details.

[0096] Then, the contrast-limited adaptive histogram equalization (CLAHE) algorithm is used for image enhancement. The CLAHE algorithm first divides the image into multiple non-overlapping small blocks, such as (8×8) in size; then, histogram equalization is performed on each small block, but the degree of contrast enhancement is limited. In this invention, the contrast limit threshold is set to 40 to avoid noise caused by over-enhancement; then, the processing results of each small block are stitched together by bilinear interpolation to improve the image clarity and detail, and obtain the final enhanced image, which is convenient for subsequent feature extraction.

[0097] Third, baseline correction and dynamic time warping (DTW) peak alignment are performed on the acquired Raman chemical fingerprint data to eliminate the effects of baseline drift and peak position differences, and to ensure data consistency.

[0098] Baseline correction is based on an adaptive smoothing algorithm (where λ = 1e5, p = 0.01, λ is the smoothing parameter, and p is the threshold parameter) to remove baseline drift interference in the spectrum; the dynamic time warping expression is as follows:

[0099] D T (i',j')=d(i',j')+min{D T (i'-1,j'),D T (i',j'-1),D T (i'-1,j'-1)};

[0100] In the formula, D T (i',j') represents the first i' of the first fingerprint sequence. ' The dynamic time-warped cumulative distance obtained when matching the first feature point with the first j' feature points of the second fingerprint sequence reflects the total cost from the beginning of the sequence to the completion of the matching of the two feature points.

[0101] d(i',j') refers to the local distance between the i'th feature point of the first fingerprint sequence and the j'th feature point of the second fingerprint sequence. In the context of fingerprint data, this local distance can be a measure of the difference between the two feature points in terms of orientation, curvature, or details, and is used to measure the cost of matching a single feature point.

[0102] D T (i'-1,j') represents the cumulative distance when backtracking to the (i'-1)th feature point of the first fingerprint sequence while retaining the j'th feature point of the second fingerprint sequence; simply put, it is the total cost of matching the first (i'-1)th feature points of the first fingerprint sequence with the first j'th feature points of the second fingerprint sequence. D T (i',j'-1) represents the cumulative distance when backtracking to the j'-1th feature point of the second fingerprint sequence while retaining the i'th feature point of the first fingerprint sequence; that is, the total cost of matching the first i'th feature point of the first fingerprint sequence with the first j'-1th feature point of the second fingerprint sequence.

[0103] D T (i'-1,j'-1) represents the cumulative distance when the current i'th and j'th feature points are also matched after the first i'-1 feature points of the first fingerprint sequence and the first j'-1 feature points of the second fingerprint sequence have been matched. This is the "cumulative cost" of diagonal backtracking.

[0104] D is calculated using a recursive method. T The (i',j') algorithm can find the optimal time alignment matching method between two fingerprint sequences, improving the comparability and accuracy of chemical fingerprint data. In practical applications, due to factors such as finger pressure, angle, or acquisition equipment, the sequence length or arrangement order of fingerprint feature points may vary. The DTW algorithm can effectively handle these problems, thereby assisting in the detection or matching of genuine and fake fingerprints.

[0105] S3. To enable effective fusion of multimodal data in subsequent models, after data collection and standardization, it is necessary to construct a targeted fusion network and decision-making mechanism. The following section will detail the cross-modal feature fusion network CMF-Net and its core components used in this invention.

[0106] This invention constructs a cross-modal feature fusion network CMF-Net, and then trains the CMF-Net based on the multimodal data collected in S1 and preprocessed in S2, extracting and dynamically weighting the multimodal features to obtain the fused feature vector.

[0107] Specifically, the cross-modal feature fusion network CMF-Net includes a spectral branch, an image branch, a chemical branch, and a gated fusion layer GFU.

[0108] (1) The spectral branch uses an improved ResNet-18 network and adds an SE attention module. The ResNet-18 network solves the gradient vanishing problem in the training process of deep neural networks through residual connections. Its output expression is:

[0109] y = F(x, {W}) + x;

[0110] In the formula, y is the output, x is the input, F(x,W) is the residual function, and W is the network parameter.

[0111] The SE attention module weights the channel features, specifically through the following process:

[0112] First, perform global average pooling:

[0113]

[0114] Among them, z c For the global pooling result of channel c, u c (i",j") represents the feature value located in the i"th row and j"th column of the feature map of the cth channel.

[0115] Then perform channel weight calculation:

[0116] s c=σ'(W2δ(W1z) c ));

[0117] Where σ' is the Sigmoid function, δ is the ReLU function, and W1 and W2 are the weights of the fully connected layer.

[0118] Finally, feature weighting is performed:

[0119]

[0120] in, The c-th channel is the weighted feature; this spectral branch can focus on the Raman spectral characteristic peaks of key components in Sichuan tangerine peel, such as hesperidin, concentrated in the 1200-1400nm wavenumber range. The SE attention module can enhance the focus on the Raman spectral characteristic peaks of hesperidin, thereby enhancing the extraction capability of important chemical components in Sichuan tangerine peel.

[0121] (2) The image branch uses the ViT-B16 model combined with the multi-scale feature pyramid (FPN) to extract local texture and global morphological features. Specifically:

[0122] The ViT-B16 model segments the image into P×P small regions, the number of which is... Each region is mapped to a vector through a linear embedding layer, and after adding positional encoding, it is input into a multi-head self-attention Transformer encoder. The encoder consists of a multi-head attention layer (MSA) and a multilayer perceptron (MLP). The output of the g-th layer is calculated as follows:

[0123] Z' g =MSA(LN(Z) g-1 ))+Z g-1 ;

[0124] Z g =MLP(LN(Z') g ))+Z' g ;

[0125] In the formula, Z g Z' represents the final output of the g-th layer. g This is an intermediate result obtained by adding the output residual of the (g-1)th layer after calculation using the multi-head self-attention mechanism MSA; Z g-1The output of the (g-1)th layer is represented by ; MSA(·) represents Multi-Head Self-Attention, which is used to capture the dependencies between different positions in the input sequence; LN(·) is the layer normalization operation, which is used to normalize the input and then perform global modeling of the image to extract overall morphological features; MLP(·) is the multilayer perceptron, which is used to perform nonlinear transformations on the input to enhance the expressive power of the model.

[0126] This invention transforms images into processable feature vectors and performs global modeling using the ViT-B16 model. Based on this, it combines a multi-scale feature pyramid (FPN) to further fuse image features at different scales, enabling the image branch to comprehensively and meticulously extract the features of Sichuan tangerine peel images, providing high-quality image information for subsequent fusion with other modalities.

[0127] Specifically, the multi-scale feature pyramid (FPN) is used to obtain feature maps F1, F2, ..., F of different layers in the ViT-B16 model. m The resolutions corresponding to each feature map are r1, r2, ..., r m By using bilinear interpolation upsampling and lateral connection to fuse features at different scales, local texture and global structural information of the image are extracted to improve the feature extraction effect of Sichuan tangerine peel images.

[0128] (3) The chemical branch employs a bidirectional long short-term memory (Bi-LSTM) network to effectively analyze the temporal information in Raman chemical fingerprint data. The Bi-LSTM network consists of a forward LSTM and a backward LSTM, which can effectively capture the temporal dependencies in the data. The dimension of the hidden layer in the Bi-LSTM is set to 128, which can simultaneously consider the forward and reverse order of the data. The fingerprint data is processed from left to right by the forward LSTM and from right to left by the backward LSTM. Finally, the outputs of the two directions are concatenated as the feature output of the chemical branch, so as to better capture the feature change trend in the chemical fingerprint spectrum and provide strong chemical feature support for the identification of genuine and counterfeit Sichuan tangerine peel.

[0129] (4) The gated fusion layer GFU dynamically weights and fuses multimodal features using the Sigmoid function, with the following formula:

[0130] ω o =σ'(W o ·[f spec ,f img ,f chem ]+b);

[0131] In the formula, ω oW represents the weight of the o-th mode, where o = 1, 2, 3 correspond to the spectral, image, and chemical modes, respectively, and σ' is the Sigmoid activation function; o This is the modal weight matrix, with dimensions 3×M, where M is the dimension of a single modal feature; [f spec ,f img ,f chem [] represents the concatenated multimodal feature vector; f spec For the spectral branch output characteristics, f img For image branch output features, f chem is the output feature of the chemical branch; b is the bias vector with dimension 3.

[0132] The formula for calculating the fused feature vector is:

[0133]

[0134] Among them, f fusion For the final fused feature vector, f o Let ω be the original feature vector of the o-th modality. The gated fusion layer can dynamically adjust the weights ω based on the importance of features from different modalities. o This enables the model to adaptively focus on modal features that contribute more to distinguishing true from false, thereby achieving effective fusion of multimodal features.

[0135] The CMF-Net cross-modal feature fusion network is trained using the multimodal data acquired in S1 and preprocessed in S2, as follows:

[0136] Step 1: Collect samples of Sichuan tangerine peel from different producing areas, including Sichuan, Fujian, and Vietnam, including genuine and counterfeit products. Specifically, there are 100 genuine samples and 30 counterfeit samples from Sichuan; 80 genuine samples and 20 counterfeit samples from Fujian; and 70 genuine samples and 10 counterfeit samples from Vietnam. These samples are then divided into a training set and a validation set at a 7:3 ratio. The training set contains 210 samples (150 genuine products and 60 counterfeit products), and the validation set contains 60 samples (40 genuine products and 20 counterfeit products). All samples have been professionally verified to ensure their authenticity and accuracy.

[0137] Step 2: This invention uses multi-task cross-entropy combined with KL divergence and L2 regularization as the loss function to ensure the model's effective learning of multimodal features and consistency between modalities.

[0138] Among them, the multi-task cross-entropy loss is... ce KL divergence loss is used for classification tasks in supervised models. kl L2 regularization loss is used to constrain the consistency of feature distributions across different modalities. L2To prevent model overfitting, this invention sets the regularization parameter β to 0.001.

[0139] The expression for the total loss function is then:

[0140] Loss = Loss ce +αLoss kl +βLoss L2 ;

[0141] Where α is the weight of the KL divergence loss, with a value of 0.1.

[0142] Step 3: Select the NAdam optimizer, set the initial learning rate to 0.001, and warm up for 10 epochs. During the warm-up phase, the learning rate increases linearly from 0 to the initial learning rate to improve the stability of model training. During training, a learning rate decay strategy is adopted. When the loss function on the validation set does not decrease for 5 consecutive epochs, the learning rate is multiplied by 0.5 to accelerate the convergence speed of the model.

[0143] Step 4: When the validation set F1-score fails to improve for 20 consecutive rounds, terminate training to avoid overfitting. Simultaneously, save the model parameters that resulted in the highest validation set F1-score as the final model.

[0144] S4. Based on Support Vector Machine (SVM) and Plattscaling, output the true and false probabilities and confidence intervals of the Sichuan tangerine peel samples, and use a dynamic threshold decision engine to determine their authenticity.

[0145] The fused feature vectors are input into a Support Vector Machine (SVM) for classification. The Plattscaling method is used to calibrate the SVM's decision output to obtain the true / false probabilities of the Sichuan tangerine peel samples. Based on the normal distribution assumption of the true / false probabilities, the confidence interval is calculated and output. In this invention, the confidence interval is 95%.

[0146] The classification threshold is adaptively adjusted by a dynamic threshold decision engine to accommodate the differences in Sichuan tangerine peel from different producing areas such as Sichuan, Fujian, and Vietnam. For samples from different producing areas, different classification thresholds are set by statistically analyzing the distribution of their authenticity probabilities. For example, the classification threshold for samples from Sichuan is set to 0.5, for samples from Fujian to 0.55, and for samples from Vietnam to 0.6. Finally, the authenticity of Sichuan tangerine peel is judged based on the authenticity probability and confidence interval.

[0147] Example 1:

[0148] Identification of genuine and counterfeit Sichuan tangerine peel by its single ingredient

[0149] First, data acquisition was performed: spectral integration time 50ms, light source intensity 100%, probe incident angle 45°. To ensure data stability and repeatability, each sample was acquired three times, and the average value was taken as the final spectral data. Images were taken using a high-definition camera with a resolution higher than 3840×2160. Five images of each sample were taken from different angles, and after ambient light compensation, they were stored in PNG format. Chemical fingerprints were obtained using HPLC technology, and the experimental conditions were in accordance with the requirements of the data acquisition section above.

[0150] The model was trained according to the parameters described above, with 100 training rounds. The final F1-score on the validation set was 0.991, as shown in Table 1.

[0151] Table 1. Parameters and results of single-component detection in Sichuan tangerine peel.

[0152] detection indicators Specific values / descriptions Online testing method Streamline testing Throughput 220 samples / minute Total number of test samples 1000 items, including 800 genuine items and 200 counterfeit items. False positive rate The number of false positives was 4, with a false positive rate of 0.4%. Power consumption ≤25W Accuracy of authenticity identification ≥99.2% (F1-score = 0.991) Improved compared to a single spectral model 12.5% ​​(86.7% accuracy for a single spectral model) Recall rate of 50 dyed tangerine peel samples ≥98.7% (49 items were detected correctly)

[0153] The experimental results in Table 1 also show that the present invention has significant advantages in identifying the authenticity of single components of Sichuan tangerine peel. By adopting a streamlined detection method, the throughput reaches 220 samples / minute, which can meet the needs of rapid batch detection.

[0154] For 1000 test samples, including 800 genuine products and 200 counterfeit products, the identification accuracy rate was ≥99.2%, with an F1-score of 0.991 and a false detection rate of only 0.4%, meaning only 4 samples were falsely detected. Figure 2 As shown, the accuracy of the method proposed in this invention is improved by 12.5% ​​compared with the 86.7% accuracy of the single spectral model, demonstrating the advantages of multimodal fusion.

[0155] The recall rate of 50 dyed tangerine peel samples is ≥98.7%, with 49 samples correctly detected. It has strong anti-interference ability and can effectively identify counterfeit products. Moreover, the power consumption is ≤25W, which is suitable for the energy requirements of practical application scenarios. Overall, the invention shows outstanding performance in terms of efficiency, accuracy, anti-interference and practicality, and is suitable for the efficient identification of genuine and counterfeit Sichuan tangerine peel.

[0156] Example 2:

[0157] Simultaneous detection of multiple components including hesperidin and nobiletin

[0158] To achieve simultaneous detection of hesperidin and nobiletin, the output layer dimension of the model was changed to 2, corresponding to the content prediction of the two components respectively. The loss function adopted was weighted MSE (weight ratio 1:0.8), where the weight of hesperidin was 1 and the weight of nobiletin was 0.8, to highlight the detection accuracy of hesperidin.

[0159] The training data used the same sample set as in Example 1. However, during training, the label for each sample included not only true / false information but also the content values ​​of hesperidin and nobiletin obtained by HPLC detection. The training run consisted of 120 rounds, and the final MAE for hesperidin content on the validation set was 0.35%, and the MAE for nobiletin content was 0.20%.

[0160] Table 2. Parameters and Results of Multi-Component Detection

[0161]

[0162] Table 2 shows that the component prediction accuracy is high: for the detection of 100 test samples, the MAE of hesperidin content is 0.31% and that of norihesperidin is 0.18%, both lower than the validation set error, indicating that the method proposed in this invention can accurately predict the content of hesperidin and norihesperidin in tangerine peel, and the detection effect is reliable.

[0163] Because two components need to be predicted simultaneously, the inference delay increases by 12% compared to Example 1, from 0.25 seconds / sample to 0.28 seconds / sample. Although there is an increase, it is still within an acceptable range, ensuring the multi-component detection function while also taking into account a certain level of detection efficiency.

[0164] A mixed test was conducted on 150 samples from multiple production areas in Sichuan, Fujian, and Vietnam. The error fluctuation was ≤±0.3%, demonstrating the good adaptability of the proposed method to Sichuan tangerine peel from different production areas. The model has strong generalization ability and can effectively cope with the challenges brought about by differences in production areas in actual testing.

[0165] Therefore, this invention adopts the above-mentioned method for detecting the authenticity of Sichuan tangerine peel based on multimodal fusion. By synergistically acquiring near-infrared spectroscopy, high-resolution images, and Raman chemical fingerprints, and combining the cross-modal feature fusion network CMF-Net and dynamic threshold decision engine, it achieves high-precision identification of authenticity with an accuracy of ≥99.2%. At the same time, the system supports mixed training of samples from multiple production areas and industrialized production line deployment, with a false detection rate of ≤0.5%. It provides an innovative solution for the anti-counterfeiting of Chinese medicinal materials and is significantly superior to existing single-modal technologies.

[0166] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for detecting the authenticity of Sichuan tangerine peel based on multimodal fusion, characterized in that, Includes the following steps: S1. Simultaneously acquire multimodal data of Sichuan tangerine peel, including near-infrared spectral data, high-resolution image data, and Raman chemical fingerprint data; S2. Preprocess the acquired multimodal data using median filtering, CLAHE image enhancement, and DTW peaking. S3. Construct a cross-modal feature fusion network CMF-Net to extract and dynamically weight and fuse multimodal features to obtain the fused feature vector; S4. Based on Support Vector Machine (SVM) and Plattscaling, output the true and false probabilities and confidence intervals of the Sichuan tangerine peel samples, and use a dynamic threshold decision engine to determine their authenticity.

2. The method for detecting the authenticity of Sichuan tangerine peel based on multimodal fusion according to claim 1, characterized in that, In S1, multimodal data of Sichuan tangerine peel were collected from the same detection area of ​​the same sample under the same detection environment. Specifically: Near-infrared spectral data of Sichuan tangerine peel were collected using a near-infrared spectrometer with a wavelength range of 1000nm-2500nm. The integration time was set to 50ms, the incident angle was controlled at 45°±5°, and the light source intensity was greater than or equal to 90%. High-definition cameras with a resolution higher than 3840×2160 were used to collect surface detail image data of Sichuan tangerine peel, including its surface texture and color characteristics; Raman fingerprint data were obtained by combining high performance liquid chromatography (HPLC) with Raman spectroscopy to reflect the chemical composition and relative content of tangerine peel.

3. The method for detecting the authenticity of Sichuan tangerine peel based on multimodal fusion according to claim 2, characterized in that, In S2, the acquired near-infrared spectral data is first processed by median filtering to remove noise interference, and the median filtering window is set to 9. Then perform regularization, the regularization formula is: Where Z represents the regularized spectral data, X represents the original spectral data, μ represents the spectral mean, and σ represents the spectral standard deviation.

4. The method for detecting the authenticity of Sichuan tangerine peel based on multimodal fusion according to claim 2, characterized in that, In S2, ambient light compensation is first performed on the surface detail image data of the collected Sichuan tangerine peel. The ambient light compensation uses the gray-world algorithm, and the expression is as follows: Where μ' is the mean gray value of the current image, H and W are the height and width of the image, I(i,j) is the original pixel value of the pixel with coordinates (i,j) in the image, and i and j represent the row index and column index of the image in the spatial dimension, respectively; definition: Where, μ l Ideal grayscale value; Then the compensated pixel value I b (i,j) is: The compensated data will be stored in PNG format; Then, the contrast-limited adaptive histogram equalization (CLAHE) algorithm is used for image enhancement. The steps of the CLAHE algorithm are as follows: Divide the image into multiple non-overlapping small blocks; Perform histogram equalization on each small block separately; The processing results of each small block are stitched together using bilinear interpolation to obtain the final enhanced image, thereby improving the image's clarity and detail.

5. The method for detecting the authenticity of Sichuan tangerine peel based on multimodal fusion according to claim 2, characterized in that, In S2, baseline correction and dynamic time warping (DTW) peak alignment processing are performed on the acquired Raman chemical fingerprint data. Baseline correction is based on an adaptive smoothing algorithm to remove baseline drift interference in the spectrum; The dynamic time warping expression is as follows: D T (i',j')=d(i',j')+min{D T (i'-1,j'),D T (i',j'-1),D T (i'-1,j'-1)}; In the formula, D T (i',j') represents the dynamic time-warped cumulative distance obtained when matching the first i' feature points of the first fingerprint sequence with the first j' feature points of the second fingerprint sequence; d(i',j') refers to the local distance between the i'th feature point of the first fingerprint sequence and the j'th feature point of the second fingerprint sequence; D T (i'-1,j') represents the cumulative distance when retaining the j'-th feature point of the second fingerprint sequence and backtracking to the i'-1-th feature point of the first fingerprint sequence; D T (i',j'-1) represents the cumulative distance when backtracking from the i'-th feature point of the first fingerprint sequence to the j'-1-th feature point of the second fingerprint sequence, while retaining the i'-th feature point of the first fingerprint sequence; D T (i'-1,j'-1) represents the cumulative distance when the current i'th and j'th feature points are matched after the first i'-1 feature points of the first fingerprint sequence and the first j'-1 feature points of the second fingerprint sequence have been matched.

6. The method for detecting the authenticity of Sichuan tangerine peel based on multimodal fusion according to claim 5, characterized in that, In S3, the cross-modal feature fusion network CMF-Net includes a spectral branch, an image branch, a chemical branch, and a gated fusion layer GFU; The spectral branch employs an improved ResNet-18 network with an added SE attention module. The ResNet-18 network addresses the vanishing gradient problem during deep neural network training through residual connections, and its output expression is: y = F(x, {W}) + x; In the formula, y is the output, x is the input, F(x,W) is the residual function, and W is the network parameter; The SE attention module weights the channel features, specifically through the following process: First, perform global average pooling: Among them, z c For the global pooling result of channel c, u c (i",j") represents the feature value located at the i"th row and j"th column position in the feature map of the cth channel; Then perform channel weight calculation: s c =s ' (W2δ(W1z c )); Where σ' is the Sigmoid function, δ is the ReLU function, and W1 and W2 are the weights of the fully connected layer; Finally, feature weighting is performed: in, This represents the weighted feature of the c-th channel.

7. The method for detecting the authenticity of Sichuan tangerine peel based on multimodal fusion according to claim 6, characterized in that, In S3, the image branch uses the ViT-B16 model combined with the multi-scale feature pyramid (FPN) to extract local texture and global morphological features. Specifically: The ViT-B16 model segments the image into P×P small regions. Each region is mapped to a vector through a linear embedding layer. After adding positional encoding, the vector is input into a multi-head self-attention Transformer encoder, which consists of a multi-head attention layer (MSA) and a multilayer perceptron (MLP). The output of the g-th layer is calculated as follows: Z' g =MSA(LN(Z g-1 ))+Z g-1 ; WITH g =MLP(LN(Z' g ))+Z' g ; In the formula, Z g Z' represents the final output of the g-th layer. g Z represents the intermediate result after calculation using the multi-head self-attention mechanism MSA, added to the output residual of the (g-1)th layer; g-1 Represents the final output of the (g-1)th layer; MSA(·) denotes the multi-head self-attention mechanism; LN(·) is the layer normalization operation; MLP(·) is the multilayer perceptron. Then, feature maps F1, F2, ..., F of different layers of the ViT-B16 model are obtained through the multi-scale feature pyramid FPN. m The resolutions corresponding to each feature map are r1, r2, ..., r m Features at different scales are fused through bilinear interpolation upsampling and lateral connectivity.

8. The method for detecting the authenticity of Sichuan tangerine peel based on multimodal fusion according to claim 7, characterized in that, In S3, the chemical branch uses a bidirectional long short-term memory network (Bi-LSTM) to analyze the temporal information in the Raman chemical fingerprint data. The Bi-LSTM network consists of a forward LSTM and a backward LSTM.

9. The method for detecting the authenticity of Sichuan tangerine peel based on multimodal fusion according to claim 8, characterized in that, In S3, the gated fusion layer GFU dynamically weights and fuses multimodal features using the Sigmoid function, as shown in the formula: oh o =σ'(W o ·[f spec ,f img ,f chem ]+b); In the formula, ω o Let W be the weight of the o-th mode, where o = 1, 2, 3 correspond to the spectral, image, and chemical modes, respectively. σ' is the Sigmoid activation function. o Let f be the modal weight matrix. spec ,f img ,f chem [] represents the concatenated multimodal feature vector; f spec For the spectral branch output characteristics, f img For image branch output features, f chem The output features are for the chemical branch, and b is the bias vector; The formula for calculating the fused feature vector is: Among them, f fusion For the final fused feature vector, f o is the original feature vector of the o-th mode.

10. The method for detecting the authenticity of Sichuan tangerine peel based on multimodal fusion according to claim 9, characterized in that, In S4, the feature vector fused from S3 is input into the support vector machine (SVM) for classification. The Plattscaling method is used to calibrate the decision output of the SVM to obtain the true and false probabilities of the Sichuan tangerine peel samples. Based on the normal distribution assumption of the true and false probabilities, the confidence interval is calculated and output. The classification threshold is adaptively adjusted by a dynamic threshold decision engine, and the true or false judgment is made based on the true or false probability and confidence interval.

Images