Cattail pollen content prediction method and system based on hyperspectral double-flow multi-scale CNN (Convolutional Neural Network)

Abstract

The invention provides a cattail pollen content prediction method and system based on a hyperspectral double-flow multi-scale CNN, and belongs to the field of quality detection of traditional Chinese medicinal materials. The method comprises the following steps: acquiring hyperspectral image data of a pollen typhae sample, preprocessing the acquired hyperspectral image data, and extracting region-of-interest spectral data of the pollen typhae sample; inputting the extracted spectral data of the region of interest into a trained double-flow multi-scale residual fusion CNN model, performing multi-scale feature extraction and fusion by using a VNIR multi-scale branch and an NIR multi-scale branch, and outputting multi-scale fusion features; inputting the multi-scale fusion features into an attention fusion module, and outputting final fusion features; and based on the final fusion features, outputting a content prediction value of the active ingredients of the pollen typhae through a content prediction layer. The method solves the problems that a traditional detection method is low in efficiency, samples are damaged, and an existing spectrum model ignores wave band characteristic differences, so that precision is insufficient, and rapid, lossless and high-precision prediction of the pollen typhae content is achieved.

Description

Technical Field

[0001] This invention belongs to the field of quality testing technology of Chinese medicinal materials, and in particular relates to a method and system for predicting the content of Typha pollen based on hyperspectral dual-stream multi-scale CNN. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Typha pollen, the dried pollen of Typha orientalis or related species in the Typhaceae family, possesses hemostatic, blood-activating, and urinary-diuretic properties. Its pharmacological activity and its active ingredients (such as isorhamnetin-3-) are key components. O The content of neohesperidin and typhain is closely related. According to relevant regulations, the total amount of the above components in Typha pollen shall not be less than 0.50%. Therefore, establishing an accurate and efficient method for content determination is crucial to ensuring the quality of Typha pollen and guaranteeing its clinical efficacy.

[0004] Currently, the determination of the effective components in Typha pollen mainly relies on traditional chemical analysis methods such as high-performance liquid chromatography (HPLC). While these methods offer high accuracy, they generally suffer from the following drawbacks: cumbersome procedures requiring complex sample extraction and purification pretreatment steps; long detection cycles, with the analysis of a single sample typically taking several hours; the need for large amounts of organic solvents, which contradicts the concept of green testing; and difficulty in achieving rapid, online analysis of large batches of samples, failing to meet the urgent needs of the modern Chinese medicine industry for real-time, in-situ quality monitoring of raw materials and finished products during production.

[0005] Hyperspectral imaging, as an emerging non-destructive testing technology, can simultaneously acquire spatial image information and continuous spectral information of the tested object, forming a three-dimensional data cube that integrates image and spectrum, providing a new technical path for rapid and non-destructive analysis of traditional Chinese medicine. However, while hyperspectral data brings rich information, it also faces challenges such as high data dimensionality, a large amount of redundant information, and complex inter-band correlations. Especially in the field of traditional Chinese medicine, different spectral ranges have different interaction mechanisms with different chemical groups in medicinal materials, and the chemical information they carry has different focuses. In existing technologies, when using hyperspectral technology combined with traditional machine learning or single-stream deep convolutional neural networks for content prediction, all band data are often simply mixed and processed, ignoring the differences in physical characteristics and information value between different band intervals. This results in insufficient feature extraction, weak discriminative power, and limited prediction accuracy and model generalization ability, making it difficult to meet the high requirements of pharmacopoeia standards for quantitative analysis. Summary of the Invention

[0006] In order to overcome the above-mentioned deficiencies of the prior art, the present application provides a high-spectrum double-flow multi-scale CNN-based caltrops content prediction method and system to solve the problems of low detection efficiency, insufficient accuracy and inability to adapt to multi-band spectral characteristic differences in the prior art.

[0007] To achieve the above object, one or more embodiments of the present application provide the following technical solutions: The present application provides a high-spectrum double-flow multi-scale CNN-based caltrops content prediction method in the first aspect; The high-spectrum double-flow multi-scale CNN-based caltrops content prediction method comprises: Collecting high-spectrum image data of caltrop samples, the high-spectrum image data comprising visible-near-infrared band data and short-wave infrared band data; Pretreating the collected high-spectrum image data, and extracting the region of interest spectral data of the caltrop samples by using a centroid-based region of interest extraction algorithm; Inputting the extracted region of interest spectral data into a trained double-flow multi-scale residual fusion CNN model to output final fusion features; wherein the trained double-flow multi-scale residual fusion CNN model uses a VNIR multi-scale branch and a NIR multi-scale branch to perform multi-scale feature extraction and fusion on the region of interest spectral data of the corresponding band, and outputs multi-scale fusion features; inputting the multi-scale fusion features into a cross-band attention fusion module for interaction and dynamic weighted fusion, and outputting final fusion features; Based on the final fusion features, outputting the content prediction value of the effective component of the caltrop by a content prediction layer.

[0008] As a further technical solution, the pretreatment of the collected high-spectrum image data comprises: The high-spectrum image data of the collected caltrop samples is subjected to black and white correction to eliminate noise interference, and the correction formula is as follows:

[0009] wherein, is the corrected reflectivity, is the original reflectivity, is a black reference image, is a white reference image.

[0010] As a further technical solution, the region of interest spectral data of the caltrop samples is extracted by using a centroid-based region of interest extraction algorithm, comprising: Extracting the three-dimensional data of the pretreated high-spectrum image and calculating the centroid coordinates; According to the center of mass coordinates, a circle is drawn, and whether the related pixel points are in the circle is judged according to the Euclidean distance. The pixel points in the circle are regarded as ROI candidate pixels, the spectral data of all ROI candidate pixels are extracted, and the wavelength is arranged in priority; The wavelength points of each batch of samples are combined together to form the spectral data of the region of interest of the sample.

[0011] As a further technical solution, the VNIR multi-scale branch and the NIR multi-scale branch each include a multi-scale feature extraction path; The multi-scale feature extraction path includes: A first path for extracting local detail features, including a 1x1 convolution layer, a batch normalization layer, a ReLU activation function layer and a max pooling layer connected in turn; A second path for extracting medium-scale correlation features, including a 1x1 convolution layer, a one-dimensional dilated convolution layer with a first dilution rate, a batch normalization layer, a ReLU activation function layer and a max pooling layer connected in turn; A third path for extracting global trend features, including a 1x1 convolution layer, a one-dimensional dilated convolution layer with a second dilution rate, a batch normalization layer, a ReLU activation function layer and a max pooling layer connected in turn.

[0012] As a further technical solution, the multi-scale feature extraction path further includes a multi-scale feature fusion unit for channel splicing the output features of the first path, the second path and the third path, and performing dynamic weight fusion through a 1x1 convolution layer to obtain multi-scale fusion features.

[0013] As a further technical solution, the cross-band attention fusion module includes: A position encoding unit for adding learnable position encoding to the input multi-scale fusion features; A multi-head self-attention unit including multiple parallel attention heads for extracting cross-band correlation features from the features after adding position encoding, and outputting attention features; A dynamic weight fusion unit for calculating the dynamic weight between the attention features output by the VNIR multi-scale branch and the NIR multi-scale branch, and performing weighted summation on the attention features of the two branches based on the dynamic weight to obtain the final fusion features.

[0014] As a further technical solution, based on the final fusion features, the content prediction value of the effective component of the cattail is output through a content prediction layer, including: The final fusion feature output by the cross-band attention fusion module is taken as an input feature of the content prediction layer; a first full connection layer operation is performed on the input final fusion feature, a feature space is mapped through a linear transformation function, then batch normalization operations are sequentially performed to reduce feature distribution differences, a ReLU activation function operation is utilized to introduce nonlinear feature expression, and a first refined feature is obtained; The first refined feature is taken as an input, a second full connection layer operation is performed, a low-dimensional feature space is further mapped through a linear transformation function, batch normalization and ReLU activation function processing are repeated, feature redundancy is eliminated and effective feature correlation is strengthened, and a second refined feature is obtained; A final linear transformation is performed on the second refined feature, a one-dimensional value is mapped from a dimensional feature through a linear transformation function, and a content prediction value of the effective component of the typhae is obtained.

[0015] The second aspect of the present application provides a typha content prediction system based on hyperspectral double-flow multi-scale CNN.

[0016] The typha content prediction system based on hyperspectral double-flow multi-scale CNN comprises: The hyperspectral data acquisition module is configured to acquire hyperspectral image data of the typha sample, wherein the hyperspectral image data comprises visible-near-infrared band data and short-wave infrared band data; The data processing module is configured to preprocess the acquired hyperspectral image data, and extract region of interest spectral data of the typha sample by using a centroid-based region of interest extraction algorithm; The final fusion feature output module is configured to input the extracted region of interest spectral data into the trained double-flow multi-scale residual fusion CNN model, and output final fusion features; wherein the trained double-flow multi-scale residual fusion CNN model performs multi-scale feature extraction and fusion on the region of interest spectral data of the corresponding band by using a VNIR multi-scale branch and a NIR multi-scale branch, and outputs multi-scale fusion features; the multi-scale fusion features are input into the cross-band attention fusion module for interaction and dynamic weighted fusion, and the final fusion features are output; The content prediction output module is configured to output a content prediction value of the effective component of the typha based on the final fusion features through a content prediction layer.

[0017] The third aspect of the present application provides a computer readable storage medium having a program stored thereon, wherein the program is executed by a processor to realize the steps in the typha content prediction method based on hyperspectral double-flow multi-scale CNN according to the first aspect of the present application.

[0018] The fourth aspect of the present application provides an electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor implements the steps in the pollenpaeonia content prediction method based on hyperspectral double-flow multi-scale CNN according to the first aspect of the present application when executing the program.

[0019] The above one or more technical solutions have the following beneficial effects: (1) The present application adopts a deep learning architecture of double-flow multi-scale residual fusion. This architecture extracts the optimal features of visible-near infrared and short-wave infrared bands through independent VNIR and NIR branches, and performs adaptive weighted fusion through cross-band attention mechanism, fully exploiting the deep physical and chemical information of different band spectra. Experimental results show that the determination coefficient (R²) on the validation set is as high as 0.9405, and the root mean square error (RMSE) is as low as 0.0359, and the prediction accuracy is significantly better than traditional spectral analysis models and single-flow deep learning models, fully meeting the strict requirements of pharmacopoeia on the accuracy of content determination. In addition, the model in the present application enhances the comprehensive perception ability of the model to local details, mid-range correlations and global trends through multi-scale feature extraction. The dynamic adaptive fusion mechanism enables the model to intelligently adjust the dependence weight of different band information according to the spectral characteristics of specific samples. After parameter optimization and a large number of sample training, the model shows stable prediction performance for pollenpaeonia samples from different producing areas and different batches, effectively overcoming the performance fluctuation problem of traditional models caused by sample differences, and has good practical application universality and reliability.

[0020] (2) The detection process of the present application only relies on the spectral information collected by the hyperspectral imaging device, without adding any chemical reagent, and without damaging the sample itself. This makes the precious medicinal material sample still available for other purposes after detection, avoiding sample loss and environmental pollution caused by chemical methods, and fully meeting the modern analysis concept of green, environmental protection and sustainability. The present application greatly shortens the detection period of traditional high performance liquid chromatography (HPLC) and does not require complex sample pretreatment. It can realize rapid and in-situ analysis of pollenpaeonia samples, greatly improving the detection throughput, providing a practical technical means for raw material acceptance, intermediate product monitoring and finished product rapid release in the production process of traditional Chinese medicine, and providing strong support for the intelligentization and automation upgrade of traditional Chinese medicine industry.

[0021] The advantages of the additional aspects of the present application will be partially given in the following description, partially will become obvious from the following description, or will be known by the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings, which form a part of the present application, are used to provide further understanding of the present application, and the illustrative embodiments of the present application and their description are used to explain the present application, and do not constitute improper limitations on the present application.

[0023] Figure 1 A flow chart of the method of the first embodiment.

[0024] Figure 2 An architecture diagram of the dual-stream multi-scale residual fusion CNN model in the first embodiment.

[0025] Figure 3 A scatter plot of the true value and the predicted value of the Pu Huang validation set data in the first embodiment.

[0026] Figure 4 A system structure diagram of the second embodiment. DETAILED DESCRIPTION

[0027] It should be noted that the following detailed description is exemplary in nature and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0028] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application.

[0029] The embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0030] Embodiment One The present embodiment discloses a Pu Huang content prediction method based on hyperspectral dual-stream multi-scale CNN. Visible-near infrared (400-1000nm) and shortwave infrared (900-1700nm) dual-band spectral data of Pu Huang samples are collected by a hyperspectral imaging system, and after preprocessing, they are input into a dual-stream adaptive feature fusion CNN model to realize accurate prediction of the content of effective components. By constructing a dual-stream feature extraction framework with professional division and intelligent collaboration and a dynamic adaptive fusion mechanism, the limitations of feature extraction of traditional single-stream models are broken through, and the efficiency, accuracy and robustness of detection are significantly improved.

[0031] Specifically, as shown in Figure 1 The Pu Huang content prediction method based on hyperspectral dual-stream multi-scale CNN includes: Step S1, collecting hyperspectral image data of Pu Huang samples, the hyperspectral image data including visible-near infrared band data and shortwave infrared band data.

[0032] In this embodiment, a hyperspectral imaging system is built to collect hyperspectral image data of the sample of pollen. The hyperspectral imaging system is composed of a MV.C VNIR camera, a NIR camera, 8 halogen tungsten lamps, a mobile platform, a white calibration plate and a computer, and the data collection is completed in a closed darkroom. The VNIR camera collects 301 data points in the 400-1000nm band with a sampling interval of 2.0nm, an exposure time of 5.0ms, and a camera height of 32cm from the mobile platform; the NIR camera collects 201 data points in the 900-1700nm band with a sampling interval of 4.0nm, an exposure time of 6.5ms, and a camera height of 37cm from the mobile platform.

[0033] After the hyperspectral imaging system is built, dark current calibration and spectral response calibration are performed. Specifically, all light sources are turned off, the camera lens cover is covered, and a dark current image is collected to eliminate camera noise; then a standard white calibration plate with a reflectivity of 95% is placed at the center of the platform, and a white reference image is collected for subsequent black and white correction of spectral data.

[0034] In step S2, the collected hyperspectral image data is preprocessed, and a centroid-based region of interest extraction algorithm is used to extract the region of interest spectral data of the pollen sample.

[0035] Due to the interference of camera dark current, light source intensity fluctuation, environmental stray light reflection and the like in the hyperspectral imaging process, the collected hyperspectral image data needs to be corrected for black and white, and the original gray value is converted into normalized reflectivity. Specifically, the black reference image and the white reference image obtained after the dark current calibration and the spectral response calibration of the hyperspectral imaging system are built are selected, and the original reflectivity of each pixel point is corrected point by point using the following formula:

[0036] wherein, is the corrected reflectivity, is the original reflectivity, is the black reference image, is the white reference image.

[0037] Since the corrected spectral data may still have random fluctuations caused by camera electronic noise, in this embodiment, moving average filtering is also used for smoothing processing, with 5 wavelength points as the window, the sliding average calculation is performed on the continuous band reflectivity curve of each pixel, the high-frequency random noise is preliminarily eliminated, and 5 wavelength points correspond to 10nm in the VNIR band and 20nm in the NIR band, which takes into account the noise reduction effect and feature preservation.

[0038] Subsequently, a centroid-based region of interest (ROI) extraction algorithm was used to obtain the spectral data of the ROI of the Typha pollen sample by spatial range filtering.

[0039] First, the three-dimensional data cube of the preprocessed hyperspectral image can be loaded using ENVI software. The resulting four-dimensional raw dataset contains spatial X coordinates, spatial Y coordinates, wavelength bands, and corresponding reflectance. The three-dimensional core data of spatial X, spatial Y, and reflectance used for ROI extraction can then be selected.

[0040] The 3D core data is divided into several data blocks according to the pixel size, and the centroid coordinates of each data block are calculated separately:

[0041]

[0042] in, This represents the total number of pixels in the current data block. and These are the x and y coordinates of the k-th point within the current data block. To ensure integer operations, the centroid is represented by integer coordinates.

[0043] Subsequently, with A circle is drawn with the centroid as the reference point and a preset radius. In this embodiment, the radius is set to 4. The point is determined based on the Euclidean distance. Whether it is inside the circle. The formula is as follows:

[0044] Pixels within the circle satisfying the above inequality are retained as candidate pixels for the Region of Interest (ROI). Complete reflectance data for all ROI candidate pixels in the 400-1000nm (VNIR band) and 900-1700nm (NIR band) wavelengths are extracted and sorted according to wavelength priority. First, reflectance data for all candidate pixels at 400nm wavelength is extracted and arranged by pixel number; then, reflectance data at 402nm wavelength is extracted and arranged by the same pixel number, and so on, until the reflectance data for all wavelengths in the VNIR band (301 wavelengths at 2nm intervals) and the NIR band (201 wavelengths at 4nm intervals) are sorted and sequentially combined to form the region of interest spectral data for the *Pyrrosia lingua* sample.

[0045] Step S3: Input the extracted spectral data of the region of interest into the trained two-stream multi-scale residual fusion CNN model, and output the final fused features. For example, Figure 2 As shown, the dual-stream multi-scale residual fusion CNN model consists of three parts: VNIR multi-scale branch, NIR multi-scale branch, and cross-band attention fusion module.

[0046] Step S31, the extracted spectral data of the region of interest of the sample is split into VNIR band data and NIR band data according to the wave band range, and after data dimension conversion and normalization processing, it is input into the corresponding VNIR multi-scale branch and NIR multi-scale branch respectively. The data dimension conversion and normalization processing are used to ensure that the input data distribution is consistent with the training data, and to avoid model prediction bias.

[0047] Step S32, the VNIR multi-scale branch and the NIR multi-scale branch have the same structure, and both complete local-mesoscale-global feature extraction and fusion through the MSRF-ACB architecture. Specifically, the multi-scale feature extraction path of the VNIR multi-scale branch and the NIR multi-scale branch includes: The first path includes a 1×1 convolution layer, a batch normalization layer, a ReLU activation function layer and a maximum pooling layer connected in turn. It is mainly used to extract local detail features and capture local peak features in the 5-7 wavelength range of the spectrum. The 1×1 convolution is used to complete the channel transformation, which reduces the amount of calculation while retaining the key local information. Then, the batch normalization and the ReLU activation function are used in turn to eliminate the distribution difference between channels and enhance the expression ability of the model to local features by introducing nonlinearity. Finally, the maximum pooling operation is performed on the activated features to downsample them and retain the maximum response value in the local area, and the local detail features are output. The formula is as follows:

[0048] wherein, is the local detail feature; is the maximum pooling operation; is the batch normalization layer; is the feature of the corresponding preprocessed spectral data; is the convolution kernel size, indicating that the kernel length of the 1×1 convolution in this path is 1; is the output channel number.

[0049] The second path includes a 1×1 convolution layer, a one-dimensional dilated convolution layer with a first expansion rate, a batch normalization layer, a ReLU activation function layer and a maximum pooling layer connected in turn. It is used for mesoscale correlation feature extraction, and the core function is to capture the mesoscale correlation features in the 10-12 wavelength range. Specifically, the features are first expanded through 1×1 convolution and then through one-dimensional dilated convolution with a first expansion rate d=2, and then batch normalization, ReLU activation function and maximum pooling are used in turn. The formula is as follows:

[0050] wherein, is the extracted mesoscale correlation feature.

[0051] The third path includes a 1x1 convolution layer, a one-dimensional dilated convolution layer with a second expansion rate, a batch normalization layer, a ReLU activation function layer, and a maximum pooling layer connected in sequence. It is used to extract the global baseline trend features in the 15-18 wavelength range in the spectrum. The technical process is as follows: after 1x1 convolution, a one-dimensional dilated convolution with a second expansion rate d=4 is performed, and then batch normalization, ReLU activation function, and maximum pooling are performed in sequence. The formula is as follows:

[0052] wherein, is the extracted global trend feature.

[0053] Further, the multi-scale feature extraction path further includes a multi-scale feature fusion unit. The channel features output by the above three paths are , , spliced, and then a 1x1 convolution is performed to learn dynamic weights, so as to realize intelligent fusion and obtain multi-scale fusion features. The specific fusion formula is as follows:

[0054] wherein, is the multi-scale fusion feature. is the output channel number after multi-scale feature fusion.

[0055] In step S33, the multi-scale fusion features are input into a cross-band attention fusion module for interaction and dynamic weighted fusion, and the final fusion features are output.

[0056] A learnable position encoding is added to the multi-scale fused features to highlight the physical meaning corresponding to the order of the spectral bands. Specifically, the multi-scale fusion features and the learnable position encoding parameter P are added element by element to obtain the features after adding the position encoding (xpos). The dimension of the position encoding parameter P is 1xLposxD, Lpos is the number of spectral bands, D=64 is the feature embedding dimension, and the initial value is a random number in the interval [-0.1, 0.1].

[0057] The 8 parallel attention heads are used to firstly capture cross-band correlation features of the features added with position encoding. Each attention head independently projects the features added with position encoding as query, key and value matrices, and performs linear transformation on the query projection matrix, the key projection matrix and the value projection matrix corresponding to the attention head. Secondly, the product of the transpose of the query projection result and the key projection result in each head is scaled, and the result is converted into attention weight in the interval of 0-1 through the Softmax function. The output features of a single attention head are obtained by weighting and multiplying the attention weight and the value projection result, as follows:

[0058] wherein, is a query vector; is a key vector; is a value vector; is a transpose.

[0059] The output features of the 8 attention heads are spliced and linearly transformed through an output projection matrix to obtain the multi-head self-attention features of the VNIR branch and the NIR branch. The dimensions of the query projection matrix, the key projection matrix and the value projection matrix corresponding to each attention head are all 64x8, and the single head dimension is 8. The dimension of the output projection matrix is 64x64.

[0060] The adaptive weighted fusion of the features of the VNIR and NIR branches is realized by calculating the attention weights of the VNIR and NIR branches, and the specific formula is as follows:

[0061]

[0062] In the formula, is a dynamic weight of the VNIR branch feature; is a dynamic weight of the NIR branch feature; is a cosine similarity function; is the multi-head self-attention feature output by the VNIR branch; is the multi-head self-attention feature output by the NIR branch; is the preliminary interaction feature of the VNIR and NIR branch features, which is used as a reference benchmark for similarity calculation.

[0063] Further, the dynamic weight of the VNIR branch feature and the dynamic weight of the NIR branch feature are weighted and fused, and the formula is as follows:

[0064] wherein, The final fusion feature obtained by weighting the double-branch features by weights.

[0065] At step S4, a content prediction value of the effective component of Typhae Pollen is output by the content prediction layer based on the final fusion feature.

[0066] The final fusion feature output by the cross-band attention fusion module is taken as the input feature of the content prediction layer. The first full connection layer operation is performed on the input final fusion feature, the feature space is mapped through a linear transformation function, then batch normalization is performed in sequence to reduce the feature distribution difference, and the ReLU activation function operation is used to introduce nonlinear feature expression, to obtain the first refined feature, as follows:

[0067] wherein, is the first refined feature; is the number of neurons of the first full connection layer; is the linear mapping function.

[0068] Secondly, taking the first refined feature as the input, the second full connection layer operation is performed, further mapped to the low-dimensional feature space through the linear transformation function, and the batch normalization and ReLU activation function processing are repeated to eliminate feature redundancy and strengthen effective feature association, to obtain the second refined feature, as follows:

[0069] wherein, is the second refined feature, is the number of neurons of the second full connection layer.

[0070] Finally, the final linear transformation is performed on the second refined feature, the dimension feature is mapped to one-dimensional numerical value through the linear transformation function, to obtain the final output content prediction value of the effective component of Typhae Pollen.

[0071]

[0072] wherein, is the final output content prediction value of the effective component of Typhae Pollen.

[0073] Finally, in order to verify the effectiveness of the method, the method is compared with the traditional high performance liquid chromatography (HPLC) method. In the model training process, the Adam optimizer is used to minimize the mean square error (MSE), and the ReduceLROnPlateau learning rate scheduler is used to avoid model overfitting and improve generalization ability. The experimental results show that the training set Rc² is 0.9448, the RMSEc is 0.0331, the validation set Rv² is 0.9405, and the RMSEv is 0.0359, which has excellent prediction accuracy. The scatter plot of the true value and the predicted value of the validation set is shown in Figure 3 The true value represents the traditional high performance liquid chromatography measurement of the content of Radix Typhonii, and it can be seen that the method used in the application is highly fitted with the data measured by the traditional high performance liquid chromatography. Compared with the traditional method, the application has faster detection speed, simple operation and can realize rapid and online analysis of a large number of samples, and has robustness and higher generalization ability.

[0074] Example Two The embodiment discloses a Radix Typhonii content prediction system based on hyperspectral dual-flow multi-scale CNN. As Figure 4 shown, the Radix Typhonii content prediction system based on hyperspectral dual-flow multi-scale CNN comprises: A hyperspectral data acquisition module is configured to acquire hyperspectral image data of Radix Typhonii samples, wherein the hyperspectral image data comprises visible-near infrared band data and short-wave infrared band data. A data processing module is configured to preprocess the acquired hyperspectral image data, and extract region of interest spectral data of the Radix Typhonii samples by using a region of interest extraction algorithm based on a centroid. A final fusion feature output module is configured to input the extracted region of interest spectral data into a trained dual-flow multi-scale residual fusion CNN model, and output final fusion features. The trained dual-flow multi-scale residual fusion CNN model uses a VNIR multi-scale branch and a NIR multi-scale branch to perform multi-scale feature extraction and fusion on the region of interest spectral data of the corresponding waveband, and outputs multi-scale fusion features. The multi-scale fusion features are input into a cross-waveband attention fusion module for interaction and dynamic weighted fusion, and the final fusion features are output. A content prediction output module is configured to output a content prediction value of an effective component of Radix Typhonii based on the final fusion features through a content prediction layer.

[0075] Example Three The purpose of the embodiment is to provide a computer readable storage medium.

[0076] A computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the method for predicting the content of pollen based on hyperspectral dual-flow multi-scale CNN according to embodiment 1.

[0077] Embodiment four The purpose of this embodiment is to provide an electronic device.

[0078] An electronic device comprising a memory, a processor, and a program stored on the memory and executable on the processor, wherein the processor implements the steps of the method for predicting the content of pollen based on hyperspectral dual-flow multi-scale CNN according to embodiment 1 when executing the program.

[0079] The steps and methods involved in the devices of embodiments two, three and four above correspond to embodiment one, and the specific embodiments can be seen in the relevant description of embodiment one. The term "computer readable storage medium" should be understood to include a single medium or multiple media of one or more instruction sets; it should also be understood to include any medium capable of storing, encoding, or carrying instruction sets for execution by a processor and causing the processor to perform any of the methods of the present application.

[0080] Those skilled in the art should understand that the above-mentioned modules or steps of the present application can be implemented by a general computer device, alternatively, they can be implemented by program codes executable by a computing device, so that they can be stored in a storage device for execution by a computing device, or they can be made into individual integrated circuit modules, or a plurality of modules or steps among them can be made into a single integrated circuit module. The present application is not limited to any specific combination of hardware and software.

[0081] Although the specific embodiments of the present application have been described above in conjunction with the accompanying drawings, it is not a limitation on the scope of protection of the present application, and those skilled in the art should understand that various modifications or variations made by those skilled in the art on the basis of the technical solutions of the present application without creative labor are still within the scope of protection of the present application.

Claims

1. A method for predicting Typhae pollen content based on hyperspectral dual-stream multi-scale CNN, characterized in that, include: Hyperspectral image data of Typha pollen samples were collected, including visible-near infrared data and short-wave infrared data. The acquired hyperspectral image data were preprocessed, and the region of interest spectral data of the cattail pollen sample was extracted using a centroid-based region of interest extraction algorithm. The extracted spectral data of the region of interest is input into a trained two-stream multi-scale residual fusion CNN model, which outputs the final fused features. The trained two-stream multi-scale residual fusion CNN model utilizes VNIR and NIR multi-scale branches to extract and fuse multi-scale features from the spectral data of the region of interest in the corresponding bands, outputting multi-scale fused features. These multi-scale fused features are then input into a cross-band attention fusion module for interactive and dynamically weighted fusion, outputting the final fused features. Based on the final fusion features, the predicted content values ​​of the effective components of Typha pollen are output through the content prediction layer.

2. The method for predicting Typhae pollen content based on hyperspectral dual-stream multi-scale CNN as described in claim 1, characterized in that, The acquired hyperspectral image data is preprocessed, including: To eliminate noise interference, the hyperspectral image data of the collected cattail pollen samples were subjected to black and white correction. The correction formula is shown below: in, The corrected reflectivity. Original reflectivity Black reference image, White reference image.

3. The method for predicting Typhae pollen content based on hyperspectral dual-stream multi-scale CNN as described in claim 1, characterized in that, A centroid-based region of interest (ROI) extraction algorithm was used to extract the spectral data of the region of interest (ROI) in the *Typha orientalis* sample, including: Extract the three-dimensional data from the preprocessed hyperspectral image and calculate the centroid coordinates; Draw a circle based on the centroid coordinates, determine whether the relevant pixels are inside the circle based on the Euclidean distance, and use the pixels inside the circle as ROI candidate pixels. Extract the spectral data of all ROI candidate pixels and sort them according to wavelength priority. The wavelength points of each batch of samples are combined sequentially to form the region of interest spectral data of the cattail pollen sample.

4. The method for predicting Typhae pollen content based on hyperspectral dual-stream multi-scale CNN as described in claim 1, characterized in that, Both the VNIR multi-scale branch and the NIR multi-scale branch include multi-scale feature extraction pathways; The multi-scale feature extraction pathway includes: The first path is used to extract local detail features, including a 1×1 convolutional layer, a batch normalization layer, a ReLU activation function layer and a max pooling layer connected in sequence; The second pathway is used to extract mesoscale correlation features, which includes a 1×1 convolutional layer, a one-dimensional dilated convolutional layer with a first dilation rate, a batch normalization layer, a ReLU activation function layer, and a max pooling layer connected in sequence. The third path, used to extract global trend features, consists of a 1×1 convolutional layer, a one-dimensional dilated convolutional layer with a second dilation rate, a batch normalization layer, a ReLU activation function layer, and a max pooling layer connected in sequence.

5. The method for predicting Typhae pollen content based on hyperspectral dual-stream multi-scale CNN as described in claim 4, characterized in that, The multi-scale feature extraction pathway also includes a multi-scale feature fusion unit, which is used to concatenate the output features of the first, second and third pathways and perform dynamic weight fusion through a 1×1 convolutional layer to obtain multi-scale fused features.

6. The method for predicting Typhae pollen content based on hyperspectral dual-stream multi-scale CNN as described in claim 1, characterized in that, The cross-band attention fusion module includes: The location encoding unit is used to add a learnable location encoding to the input multi-scale fused features; The multi-head self-attention unit includes multiple parallel attention heads, which are used to extract cross-band correlated features from the features after adding positional encoding and output attention features. The dynamic weight fusion unit is used to calculate the dynamic weight between the attention features output by the VNIR multi-scale branch and the NIR multi-scale branch, and to perform a weighted summation of the attention features of the two branches based on the dynamic weight to obtain the final fused feature.

7. The method for predicting Typhae pollen content based on hyperspectral dual-stream multi-scale CNN as described in claim 1, characterized in that, Based on the final fusion features, the predicted content values ​​of the effective components of Typha pollen are output through the content prediction layer, including: The final fused features output by the cross-band attention fusion module are used as the input features of the content prediction layer. The first fully connected layer operation is performed on the input final fused features. The feature space is mapped by a linear transformation function, and then batch normalization is performed to reduce the feature distribution difference. The ReLU activation function is used to introduce non-linear feature expression to obtain the first refined features. Using the features refined in the first step as input, a second fully connected layer operation is performed. The linear transformation function is used to further map the features to a low-dimensional feature space. Batch normalization and ReLU activation function are repeated to eliminate feature redundancy and strengthen the association of effective features, resulting in the features refined in the second step. A final linear transformation is performed on the features after the second refinement. The dimensional features are mapped to one-dimensional values ​​through the linear transformation function, and the final output is the predicted content of the effective components of Typha pollen.

8. A system for predicting Typhae pollen content based on hyperspectral dual-stream multi-scale CNN, characterized in that: include: The hyperspectral data acquisition module is configured to acquire hyperspectral image data of the cattail pollen sample, wherein the hyperspectral image data includes visible-near infrared band data and shortwave infrared band data. The data processing module is configured to: preprocess the acquired hyperspectral image data and extract the region of interest spectral data of the cattail pollen sample using a centroid-based region of interest extraction algorithm; The final fusion feature output module is configured to: input the extracted spectral data of the region of interest into a trained dual-stream multi-scale residual fusion CNN model, and output the final fusion features; wherein, the trained dual-stream multi-scale residual fusion CNN model uses VNIR multi-scale branches and NIR multi-scale branches to perform multi-scale feature extraction and fusion on the spectral data of the region of interest in the corresponding band, and outputs multi-scale fusion features; the multi-scale fusion features are input into the cross-band attention fusion module for interactive and dynamic weighted fusion, and output the final fusion features; The content prediction output module is configured to output the content prediction value of the effective components of Typha pollen through the content prediction layer based on the final fusion features.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the method for predicting cattail pollen content based on hyperspectral dual-stream multi-scale CNN as described in any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the method for predicting cattail pollen content based on hyperspectral dual-stream multi-scale CNN as described in any one of claims 1-7.

Images