Traditional Chinese medicine image recognition method and system based on deep learning

Abstract

The application relates to a traditional Chinese medicinal material image recognition method and system based on deep learning. The method comprises the following steps: acquiring multi-modal images of traditional Chinese medicinal materials, performing registration and alignment processing on the multi-modal images to obtain a cross-modal alignment image set; generating a multi-scale feature vector based on the cross-modal alignment image set and a cross-modal feature extraction network model; performing modal decoupling and gate fusion processing on the multi-scale feature vector to obtain a fusion feature vector; performing fine-grained classification based on the fusion feature vector and a prototype contrast loss function, and outputting the corresponding category of the traditional Chinese medicinal material. The method can improve the accuracy and reliability of traditional Chinese medicinal material classification, and effectively overcome the problems of insufficient single-modal data information and one-sided feature extraction.

Description

Technical Field

[0001] This invention belongs to the field of image recognition technology, and in particular relates to a method and system for image recognition of Chinese medicinal materials based on deep learning. Background Technology

[0002] With the rapid development of image recognition technology, intelligent recognition methods based on deep convolutional neural networks (CNNs) have emerged in the field of traditional Chinese medicine (TCM) image recognition. These technologies improve the efficiency of TCM identification by automatically extracting image features and classifying them. Among these methods, advanced techniques such as multimodal fusion and attention mechanisms are widely used to handle the complex morphological features of TCM.

[0003] In traditional techniques, the identification of Chinese medicinal materials mainly adopts the following processing methods: For microstructure identification, local features are usually extracted using convolutional kernels of fixed scale (such as 3×3 or 7×7), and feature dimensionality reduction is achieved by combining pooling operations; In terms of multimodal data processing, the common practice is to directly stitch together or weightedly fuse image features of different modalities; In the classification stage, the standard cross-entropy loss function is often used for model optimization.

[0004] However, current identification methods or traditional approaches have the following key problems: In terms of microscale identification, fixed-scale convolutional kernels are unable to capture long-range periodic texture features (such as the micron-level difference in the spacing of vascular rings between ginseng and American ginseng), resulting in insufficient fine-grained distinguishing ability for similar medicinal materials; In multimodal data processing, due to the lack of an effective cross-modal alignment mechanism, the measurement of different modal features in Hilbert space is inconsistent (cosine similarity fluctuation > 0.35), affecting the feature fusion effect; Traditional loss functions cannot effectively handle fine-grained classification problems with large intra-class differences and small inter-class differences, resulting in a persistently high misidentification rate for similar medicinal materials. Summary of the Invention

[0005] Therefore, it is necessary to provide a deep learning-based image recognition method and system for Chinese medicinal materials that can improve the accuracy of Chinese medicinal material identification, in order to address the aforementioned technical problems.

[0006] Firstly, this application provides a deep learning-based method for recognizing images of traditional Chinese medicinal materials, including:

[0007] Multimodal images of Chinese medicinal materials are acquired, and the multimodal images are registered and aligned to obtain a cross-modal aligned image set.

[0008] Multi-scale feature vectors are generated based on cross-modal aligned image sets and cross-modal feature extraction network models;

[0009] Modal decoupling and gated fusion processing are performed on the multi-scale feature vectors to obtain the fused feature vectors;

[0010] Fine-grained classification is performed based on fused feature vectors and prototype contrast loss function to output the category corresponding to Chinese medicinal materials.

[0011] In one embodiment, fine-grained classification is performed based on the fused feature vector and the prototype contrastive loss function to output the categories of Chinese medicinal materials, including:

[0012] The prototype centers of each medicinal material category are initialized using the exponential translation method to obtain the prototype centers;

[0013] Filter similar hard-to-bear sample sets based on feature similarity;

[0014] Based on the fused feature vector, prototype center set, and hard-to-bear sample set, the prototype contrastive loss is calculated.

[0015] The prototype center is dynamically updated based on prototype contrast loss and exponential moving average method;

[0016] Based on the updated prototype center and fused feature vector, the category of Chinese medicinal materials is output.

[0017] In one embodiment, the update formula for the prototype center is expressed by the following formula:

[0018]

[0019] in, For the present The prototype center, For the updated The prototype center, Category in the current batch The sample set, The momentum coefficient, For the first The fusion feature vector of each sample.

[0020] In one embodiment, multimodal images of Chinese medicinal materials are acquired, and the multimodal images are registered and aligned to obtain a cross-modal aligned image set, including:

[0021] Visible light images, near-infrared images, and microscopic images of Chinese medicinal materials are acquired to form a multimodal image set;

[0022] Affine transform-wavelet registration coupling processing is performed on the multimodal image set to generate a cross-modal aligned image set; the error function in affine transform-wavelet registration is expressed by the following formula:

[0023]

[0024] in, For the image to be registered, For reference image, For Haar wavelet decomposition, Denotes the affine transformation matrix. The coefficients of the total variation regularization term, This is the smoothing constraint term for the transformation matrix.

[0025] In one embodiment, the cross-modal feature extraction network model includes a macroscopic branch and a microscopic branch that operate in parallel; the macroscopic branch is used to process visible light and near-infrared images through non-local dilated convolutional blocks to extract macroscopic morphological features; the microscopic branch is used to process microscopic images through a periodic-sensitive attention module to extract periodic microstructural features.

[0026] In one embodiment, the multi-scale feature vectors are subjected to modal decoupling and gated fusion processing to obtain fused feature vectors, including:

[0027] Apply orthogonal projection loss to the multi-scale feature vectors to obtain the decoupled multi-scale features;

[0028] Based on the gated cross-attention model, the decoupled multi-scale features are fused to obtain the fused feature vector.

[0029] In one embodiment, the prototype contrastive loss function is optimized for intra-class to inter-class distance using the following formula:

[0030]

[0031] in, For the first The fused feature vector of each sample, For the first The fused feature vector of each sample, For cosine similarity, As the prototype center of the category, For a set of samples of the same type that are difficult to bear, This is the temperature coefficient.

[0032] Secondly, this application also provides a deep learning-based image recognition system for traditional Chinese medicinal materials, comprising:

[0033] The image cross-modal alignment module is used to acquire multimodal images of Chinese medicinal materials, perform registration and alignment processing on the multimodal images, and obtain a cross-modal aligned image set;

[0034] The multi-scale feature vector module is used to generate multi-scale feature vectors based on cross-modal aligned image sets and cross-modal feature extraction network models.

[0035] The feature fusion module is used to perform modal decoupling and gated fusion processing on multi-scale feature vectors to obtain fused feature vectors.

[0036] The Chinese medicinal herb category output module is used to perform fine-grained classification based on fused feature vectors and prototype contrast loss function, and output the category of Chinese medicinal herbs.

[0037] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in the first aspect.

[0038] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect.

[0039] The aforementioned deep learning-based image recognition method and system for Chinese medicinal materials acquires and aligns multimodal images of Chinese medicinal materials, constructing a cross-modal image set with a unified spatial benchmark, effectively solving the spatial difference problem between different modal data. Utilizing a cross-modal feature extraction network model, it captures image features from multiple scales and angles, generating rich feature vectors and fully mining the potential information of each modality. Through modal decoupling and gated fusion, it accurately separates and fuses features from different modalities, ensuring that the fused feature vector contains unique information from each modality, achieving complementary advantages among multiple modalities and improving the completeness and accuracy of feature expression. Fine-grained classification based on a prototype contrastive loss function strengthens the clustering of features of similar samples and the distinguishability between different categories. Overall, it improves the accuracy and reliability of Chinese medicinal material classification, effectively overcoming problems such as insufficient information in single-modal data and one-sided feature extraction. Attached Figure Description

[0040] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 This is a flowchart illustrating a deep learning-based image recognition method for Chinese medicinal materials in one embodiment.

[0042] Figure 2 This is a schematic diagram of the structure of a deep learning-based image recognition system for Chinese medicinal materials in one embodiment;

[0043] Figure 3 This is a schematic diagram of a computer device for a deep learning-based image recognition method for Chinese medicinal materials, as shown in one embodiment. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0045] In one embodiment, such as Figure 1 As shown, a deep learning-based image recognition method for traditional Chinese medicinal materials is provided. This embodiment illustrates the application of this method to a traditional Chinese medicinal material recognition terminal. It is understood that this method can also be applied to a server, or to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0046] S101, acquire multimodal images of Chinese medicinal materials, perform registration and alignment processing on the multimodal images, and obtain a cross-modal aligned image set.

[0047] For example, various image acquisition devices, such as RGB cameras, near-infrared cameras, and hyperspectral imagers, are used to acquire multimodal images of Chinese medicinal materials; the acquired multimodal images are then transmitted to a Chinese medicinal material identification terminal; due to differences in acquisition devices, time, or viewing angles, multimodal images may have spatial and temporal deviations; registration algorithms based on feature point matching and deep learning registration networks are needed to process the multimodal images to make them consistent in spatial coordinates; after registration and alignment processing, a cross-modal aligned image set is obtained.

[0048] S102 generates multi-scale feature vectors based on cross-modal aligned image sets and cross-modal feature extraction network models.

[0049] For example, the Chinese medicinal herb identification terminal inputs a cross-modal aligned image set into a cross-modal feature extraction network model; this model can be composed of different network architectures such as convolutional neural networks and recurrent neural networks. By performing multi-scale feature extraction on the cross-modal aligned image set, feature vectors of different levels and granularities can be obtained. For instance, shallow features can reflect information such as the texture and color of the Chinese medicinal herbs, while deep features can reflect their essential features such as shape and tissue structure. Employing multi-scale feature extraction allows for the comprehensive capture of feature information of Chinese medicinal herbs at different scales, enhancing the cross-modal feature extraction network model's ability to express the features of Chinese medicinal herbs.

[0050] S103 performs modal decoupling and gated fusion processing on the multi-scale feature vectors to obtain the fused feature vectors.

[0051] For example, the Chinese medicinal herb identification terminal performs modal decoupling and gated fusion processing on multi-scale feature vectors. Modal decoupling primarily removes redundant information from different modal features, thereby highlighting the unique characteristics of each modality. Gated fusion adaptively assigns weights to different modal features based on their importance in the classification task. Through modal decoupling and gated fusion processing, the advantages of different modal features can be fully utilized, effectively avoiding the limitations of a single modality, ultimately resulting in a more representative and discriminative fused feature vector.

[0052] S104 performs fine-grained classification based on fused feature vectors and prototype contrast loss function, and outputs the category corresponding to Chinese medicinal materials.

[0053] For example, the Chinese medicinal herb identification terminal performs fine-grained classification based on fused feature vectors and a prototype contrast loss function, outputting the corresponding category of the Chinese medicinal herb through a classifier. The prototype contrast loss function aims to make the feature vectors of similar Chinese medicinal herbs as close as possible to their class prototype in the feature space, while maintaining sufficient distance from prototypes of other categories. Using the prototype contrast loss function can effectively solve the classification difficulty caused by the similarity in appearance between Chinese medicinal herbs; the prototype contrast learning method can improve the model's ability to distinguish fine-grained differences among Chinese medicinal herbs.

[0054] In the aforementioned deep learning-based image recognition method for Chinese medicinal materials, comprehensive capture of medicinal material features is achieved through multimodal image registration and alignment. A multi-scale feature extraction network model enhances the expressive power of features at different levels. Modal decoupling and gated fusion processing optimize the feature fusion method, fully leveraging the complementary advantages of each modality. Fine-grained classification based on the prototype contrast loss function effectively improves the ability to distinguish between similar-looking Chinese medicinal materials. In summary, this method improves the accuracy and robustness of Chinese medicinal material recognition while reducing dependence on single-modal data.

[0055] Optionally, fine-grained classification based on fused feature vectors and prototype contrastive loss function to output the category of Chinese medicinal materials also includes the following steps:

[0056] S1001, initialize the prototype center of each medicinal material category using the exponential translation method to obtain the prototype center.

[0057] For example, the prototype center for each medicinal herb category is initialized using the exponential translation method. The exponential translation method uses a weighted average of the feature vectors of each category in the training data, making the initial prototype center closer to the true distribution center. Specifically, the exponential translation method assigns smaller weights to earlier samples and larger weights to more recent samples, thereby dynamically adjusting the prototype center position and reducing the influence of outliers. This initialization method allows the prototype center to converge to the core region of the category features more quickly.

[0058] S1002, selects a set of similar, difficult-to-bear samples based on feature similarity.

[0059] For example, by calculating feature similarity, difficult-to-negative samples similar to the current sample's features are selected from different categories. Traditional contrastive learning typically selects negative samples randomly, but in fine-grained classification of Chinese medicinal materials, varieties with similar appearances (such as ginseng and codonopsis) require more stringent differentiation. The construction of the difficult-to-negative sample set focuses on these easily confused sample pairs. By calculating cosine similarity or Euclidean distance, samples that are closest in features to the current sample but belong to different categories are selected, enhancing the model's ability to distinguish subtle differences.

[0060] S1003 calculates the prototype contrastive loss based on the fused feature vector, prototype center set, and hard-to-bear sample set.

[0061] For example, a prototype contrastive loss is calculated based on the fused feature vectors, prototype center set, and hard-to-bear sample set. This loss function encourages feature vectors of the same class to move closer to their prototype centers while moving away from prototypes of other classes and hard-to-bear samples. Specifically, the loss function consists of two parts: an intra-class closing term (the distance between the current sample and its own class prototype) and an inter-class pushing term (the distance between the current sample and prototypes of other classes and hard-to-bear samples). By maximizing inter-class differences and minimizing intra-class differences, the model's classification ability is improved.

[0062] S1004, dynamically updates the prototype center based on prototype contrast loss and exponential moving average method.

[0063] For example, the prototype center is dynamically updated using the exponential moving average (EMA) method based on the calculated prototype contrastive loss. EMA smoothly updates the prototype position by weighting the historical prototypes and the current optimization direction, avoiding drastic fluctuations caused by noisy samples. The updated prototype center can be continuously optimized as training progresses, adapting to changes in data distribution, and is particularly suitable for classification tasks involving natural variations, such as those involving traditional Chinese medicine.

[0064] S1005, based on the updated prototype center and fused feature vector, outputs the category of Chinese medicinal materials.

[0065] For example, based on the updated prototype centers and fused feature vectors, the similarity (e.g., cosine similarity) between the sample and each prototype center is calculated, and the sample is classified into the category with the highest similarity. Through dynamically updated prototype centers, the model can more accurately capture the feature distribution of Chinese medicinal materials categories, thereby improving the accuracy of fine-grained classification, especially for varieties with similar morphology (e.g., different regional varieties of wolfberry).

[0066] Optionally, the update formula for the prototype center is expressed by the following formula:

[0067]

[0068] in, For the present The prototype center, For the updated The prototype center, Category in the current batch The sample set, The momentum coefficient, For the first The fusion feature vector of each sample.

[0069] In the above prototype center update formula, As a historical information retention item, it retains most of the information of the prototype center at previous moments, reflecting the memory characteristic of EMA for historical states; For the current batch information item, it is updated based on the average of the feature vectors of all samples in the current batch category, with the momentum coefficient complement being... The weighting of information from the current batch is controlled. EMA avoids drastic fluctuations in the prototype center by weighted averaging of historical information and current data. For small-batch training, it reduces the impact of variance between batches. When the number of samples in different categories differs significantly, EMA prevents the prototype of a few categories from being over-updated, with each batch only making small adjustments to the prototype, ensuring long-term stability. In the scenario of Chinese medicinal material identification, there may be samples from different sources or growing conditions. EMA allows the prototype center to gradually adapt to these distribution changes, improving the robustness of Chinese medicinal material identification and classification. When the momentum coefficient is large, the prototype updates slowly, is less sensitive to the current batch data, and is more resistant to noise, but may lead to slower convergence. When the momentum coefficient is small, the prototype updates quickly, pays more attention to the information of the current batch, and adapts rapidly to new data, but may introduce more fluctuations.

[0070] Optionally, acquiring multimodal images of Chinese medicinal materials, performing registration and alignment processing on the multimodal images to obtain a cross-modal aligned image set, further includes the following steps:

[0071] S2001, acquire visible light images, near-infrared images and microscopic images of Chinese medicinal materials to form a multimodal image set.

[0072] Visible light images can reflect the surface color, texture, and macroscopic morphological characteristics of medicinal materials, and are important for identifying medicinal materials with obvious color differences (such as wolfberry and honeysuckle); near-infrared images are based on the absorption characteristics of near-infrared light by the chemical components of medicinal materials, revealing the distribution of internal components (such as polysaccharides and flavonoids), and helping to identify medicinal materials that look similar but have different components; microscopic images magnify the microscopic structure of medicinal materials (such as cell morphology and crystal characteristics), providing key evidence for identifying confused varieties (such as ginseng and American ginseng); visible light images, near-infrared images, and microscopic images constitute the multimodal images of Chinese medicinal materials.

[0073] S2002, performs affine transformation-wavelet registration coupling processing on the multimodal image set to generate a cross-modal aligned image set; the error function in affine transformation-wavelet registration is expressed by the following formula:

[0074]

[0075] in, For the image to be registered, For reference image, For Haar wavelet decomposition, Denotes the affine transformation matrix. The coefficients of the total variation regularization term, This is the smoothing constraint term for the transformation matrix.

[0076] Among them, affine transformation maps images of different modalities to the same coordinate system through linear transformations such as rotation, translation, and scaling; wavelet decomposition uses Haar wavelets to decompose images into different frequency components, capture multi-scale features, and improve registration accuracy.

[0077] For example, Haar wavelet multi-scale decomposition is performed on multimodal images to obtain low-frequency approximation coefficients and high-frequency detail coefficients. A feature pyramid is constructed based on the wavelet coefficients, and affine transformation parameters are estimated at different scales. By minimizing the error function, the affine transformation matrix is ​​iteratively optimized. The optimized transformation matrix is ​​applied to the image to be registered to achieve alignment with the reference image, resulting in a cross-modal aligned image set.

[0078] In the error function of the above affine transformation-wavelet registration, It is used to measure the difference between the wavelet decomposition of the reference image and the image to be registered, and to capture the multi-scale feature matching degree. For total variation regularization coefficients, used to balance registration accuracy and noise robustness; This is a smoothing constraint term in the transformation matrix, used to control the smoothness of the transformation and avoid transformation distortion caused by overfitting. Using affine transformation-wavelet decomposition registration coupling can solve the spatial inconsistency problem caused by differences in acquisition methods between different modal images, improving robustness to morphological variations and imaging condition changes in Chinese medicinal materials.

[0079] Optionally, the cross-modal feature extraction network model includes a macro-branch and a micro-branch that work in parallel; the macro-branch is used to process visible light and near-infrared images through non-local dilated convolutional blocks to extract macro-morphological features; the micro-branch is used to process microscopic images through a periodic-sensitive attention module to extract periodic microstructural features.

[0080] For example, the cross-modal feature extraction network model adopts a dual-branch parallel architecture, capturing the macroscopic and microscopic features of Chinese medicinal materials through differentiated design. The macroscopic branch processes visible light and near-infrared images through nonlocal dilated convolution blocks: nonlocal operations capture long-range dependencies, enhancing the perception of overall morphology and texture; dilated convolution expands the receptive field without increasing the number of parameters, effectively extracting macroscopic features such as leaf shape and surface texture, while fusing chemical component distribution information from near-infrared images. The microscopic branch uses a periodic-sensitive attention module to process microscopic images: it identifies periodic microstructures such as cell wall arrangement and crystal distribution through Fourier transform and other methods, and the attention mechanism automatically focuses on periodic patterns with discriminative power (such as the layered structure of starch grains). The dual branches work together, enabling the cross-modal feature extraction network model to possess both "macroscopic perspective" and "microscopic insight" capabilities, improving the ability to distinguish between Chinese medicinal materials with similar appearances but different microstructures (such as ginseng and American ginseng).

[0081] Optionally, performing modal decoupling and gated fusion processing on the multi-scale feature vectors to obtain fused feature vectors further includes the following steps:

[0082] S3001 applies orthogonal projection loss to the multi-scale feature vectors to obtain the decoupled multi-scale features.

[0083] For example, orthogonal projection loss is used to decouple information between modalities, aiming to eliminate redundancy between features of different modalities while retaining unique information. Specifically, the Chinese herbal medicine identification terminal imposes constraints on multi-scale feature vectors from macroscopic branches (visible light / near-infrared) and microscopic branches (microscopic images), forcing the feature representations of different modalities to be orthogonal to each other in the feature space. For example, if the visual modality and the microscopic modality are feature vectors of two modalities, the orthogonal projection loss of the feature vectors of the two modalities is calculated, and minimizing this loss makes the inner product of the features of the two modalities approach zero. The orthogonal projection loss decoupling mechanism effectively separates modal shared information (such as color and texture) and modality-specific information (such as chemical composition distribution and cell structure), avoiding interference from redundant features in classification decisions and enhancing the specificity of feature representations.

[0084] S3002, based on the gated cross-attention model, fuses decoupled multi-scale features to obtain a fused feature vector.

[0085] For example, the gated cross-attention model adaptively fuses decoupled multi-scale features to achieve complementary advantages. The gated cross-attention model first calculates the attention weights between features of different modalities, and then generates a fused feature vector through weighted summation. Specifically, for macroscopic and microscopic features, gating weights are generated through an attention function, representing the "contribution" of microscopic features to macroscopic features; for example, when identifying honeysuckle, the weight of microscopic features (glandular hair density) is higher than that of macroscopic features (color). The gated cross-attention model can automatically adjust the dependencies between modalities according to sample characteristics; for example, it pays more attention to microscopic features for seed-type medicinal materials and relies more on macroscopic morphology for whole-herb medicinal materials, thereby improving the accuracy of fine-grained classification.

[0086] Optionally, the prototype contrastive loss function is optimized for intra-class to inter-class distance using the following formula:

[0087]

[0088] in, For the first The fused feature vector of each sample, For the first The fused feature vector of each sample, For cosine similarity, As the prototype center of the category, For a set of samples of the same type that are difficult to bear, This is the temperature coefficient.

[0089] In the above formula for optimizing inter-class distance using the prototype contrastive loss function, The intra-class convergence term maximizes the similarity between a sample and its own class prototype, encouraging samples of the same class to cluster in the feature space. The inter-class distance term minimizes the similarity between a sample and a hard-to-bear sample, especially samples that are similar in appearance to the current sample but not of the same class (such as ginseng and codonopsis). By taking the logarithmic ratio of the intra-class convergence term and the inter-class distance term, the joint optimization of intra-class compactness and inter-class separation is achieved.

[0090] In the aforementioned deep learning-based image recognition method for traditional Chinese medicine, the fine-grained classification performance is improved through a collaborative design of multimodal fusion, bi-branch feature extraction, orthogonal decoupling, and prototype contrastive learning. Multimodal registration, through affine transformation-wavelet coupling combined with total variational regularization and transform smoothing constraints, effectively solves the spatial alignment problem between different imaging modalities, preserving complementary information such as color texture in visible light images, component distribution in near-infrared images, and cellular structure in microscopic images. In the bi-branch network, the nonlocal dilated convolution of the macroscopic branch captures long-distance dependencies, while the periodic-sensitive attention of the microscopic branch focuses on periodic structures such as cell wall arrangement, achieving multi-scale feature extraction from millimeters to micrometers. Modal decoupling and gated fusion eliminate feature redundancy through orthogonal projection loss and dynamically allocate modal weights using cross-attention, enabling the model to adaptively adjust its dependence on different modalities based on sample characteristics. Prototype contrastive learning, through hard-burden sample mining and dynamic updating of prototype centers, constructs a compact and distinguishable category representation in the feature space, enhancing the ability to distinguish between similar-looking varieties (such as ginseng / American ginseng). In summary, this method improves classification accuracy, effectively handles fine-grained classification problems with large intra-class differences and small inter-class differences, and reduces identification errors for easily confused categories (such as Fritillaria).

[0091] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0092] Based on the same inventive concept, this application also provides a deep learning-based image recognition system for Chinese medicinal materials, which implements the aforementioned deep learning-based image recognition method for Chinese medicinal materials. The solution provided by this system is similar to the implementation scheme described in the above method. Therefore, the specific limitations of one or more deep learning-based image recognition system embodiments provided below can be found in the limitations of the deep learning-based image recognition method for Chinese medicinal materials described above, and will not be repeated here.

[0093] In one embodiment, such as Figure 2 As shown, a deep learning-based image recognition system for Chinese medicinal materials 200 is provided, comprising:

[0094] The image cross-modal alignment module 201 is used to acquire multimodal images of Chinese medicinal materials, perform registration and alignment processing on the multimodal images, and obtain a cross-modal aligned image set;

[0095] The multi-scale feature vector module 202 is used to generate multi-scale feature vectors based on cross-modal aligned image sets and cross-modal feature extraction network models.

[0096] The feature fusion module 203 is used to perform modal decoupling and gated fusion processing on multi-scale feature vectors to obtain fused feature vectors.

[0097] The Chinese medicinal material category output module 204 is used to perform fine-grained classification based on the fused feature vector and the prototype contrast loss function, and output the category of Chinese medicinal materials.

[0098] Furthermore, the Chinese medicinal materials category output module 204 includes:

[0099] The initialization module is used to initialize the prototype center of each medicinal material category using the exponential translation method, thereby obtaining the prototype center;

[0100] Filtering hard-to-bear sample sets is used to filter similar hard-to-bear sample sets based on feature similarity.

[0101] The prototype contrast loss calculation module is used to calculate the prototype contrast loss based on the fused feature vector, prototype center set, and hard-to-bear sample set.

[0102] Update the prototype center module to dynamically update the prototype center based on prototype contrast loss and exponential moving average method;

[0103] The Chinese medicinal herb category identification module is used to output the category of Chinese medicinal herbs based on the updated prototype center and fused feature vector.

[0104] Furthermore, the update formula for the prototype center is expressed by the following formula:

[0105]

[0106] in, For the present The prototype center, For the updated The prototype center, Category in the current batch The sample set, The momentum coefficient, For the first The fusion feature vector of each sample.

[0107] Furthermore, the image cross-modal alignment module 201 includes:

[0108] The multimodal image set module is used to acquire visible light images, near-infrared images, and microscopic images of Chinese medicinal materials to form a multimodal image set;

[0109] The cross-modal image alignment module is used to perform affine transformation-wavelet registration coupling processing on a multimodal image set to generate a cross-modal aligned image set; the error function in affine transformation-wavelet registration is expressed by the following formula:

[0110]

[0111] in, For the image to be registered, For reference image, For Haar wavelet decomposition, Denotes the affine transformation matrix. The coefficients of the total variation regularization term, This is the smoothing constraint term for the transformation matrix.

[0112] Furthermore, the cross-modal feature extraction network model includes a macroscopic branch and a microscopic branch that work in parallel; the macroscopic branch is used to process visible light and near-infrared images through non-local dilated convolutional blocks to extract macroscopic morphological features; the microscopic branch is used to process microscopic images through a periodic-sensitive attention module to extract periodic microscopic structural features.

[0113] Furthermore, the feature fusion module 203 includes:

[0114] The multi-scale feature decoupling module is used to apply orthogonal projection loss to the multi-scale feature vectors to obtain the decoupled multi-scale features.

[0115] The feature vector fusion module is used to fuse decoupled multi-scale features based on the gated cross-attention model to obtain a fused feature vector.

[0116] Furthermore, the prototype contrastive loss function is optimized for intra-class to inter-class distance using the following formula:

[0117]

[0118] in, For the first The fused feature vector of each sample, For the first The fused feature vector of each sample, For cosine similarity, As the prototype center of the category, For a set of samples of the same type that are difficult to bear, This is the temperature coefficient.

[0119] In one embodiment, such as Figure 3 A computer device 300 is provided, comprising:

[0120] At least one processor 301, and at least one memory 302 communicatively connected to said processor 301; said memory stores application code executable by said processor, said application code being executed by said processor to enable said processor to perform the steps of a deep learning-based Chinese medicinal herb image recognition as described above;

[0121] The computer device may also include: sensor 303; processor 301, memory 302 and sensor 303 can be connected via bus 304 or other means, the figure shows an example via bus 304. Figure 3 The character is represented by a single thick line, but this does not mean that there is only one bus or a type of bus.

[0122] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0123] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The components described as separate parts may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0124] The above-described embodiments are merely illustrative of several implementation methods of the embodiments of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the embodiments of this application, and these modifications and improvements all fall within the protection scope of the embodiments of this application.

Claims

1. A method for recognizing images of traditional Chinese medicinal materials based on deep learning, characterized in that, The method includes: Multimodal images of Chinese medicinal materials are acquired, and the multimodal images are registered and aligned to obtain a cross-modal aligned image set. Based on the cross-modal aligned image set and the cross-modal feature extraction network model, multi-scale feature vectors are generated; The multi-scale feature vectors are subjected to modal decoupling and gated fusion processing to obtain fused feature vectors; Fine-grained classification is performed based on the fused feature vector and the prototype contrast loss function to output the category corresponding to the Chinese medicinal material; The process of acquiring multimodal images of Chinese medicinal materials, performing registration and alignment processing on the multimodal images to obtain a cross-modal aligned image set includes: Visible light images, near-infrared images, and microscopic images of Chinese medicinal materials are acquired to form a multimodal image set; The multimodal image set is subjected to affine transformation-wavelet registration coupling processing to generate a cross-modal aligned image set; the error function in the affine transformation-wavelet registration is expressed by the following formula: ； in, For the image to be registered, For reference image, For Haar wavelet decomposition, Denotes the affine transformation matrix. The coefficients of the total variation regularization term, This is the smoothing constraint term for the transformation matrix.

2. The method according to claim 1, characterized in that, The fine-grained classification based on the fused feature vector and prototype contrast loss function outputs the categories of Chinese medicinal materials, including: The prototype centers of each medicinal material category are initialized using the exponential translation method to obtain the prototype centers; Filter similar hard-to-bear sample sets based on feature similarity; Based on the fused feature vector, the prototype center set composed of the prototype centers, and the hard-to-bear sample set, the prototype contrast loss is calculated. The prototype center is dynamically updated based on the prototype contrast loss and exponential moving average method. Based on the updated prototype center and fused feature vector, the category of Chinese medicinal materials is output.

3. The method according to claim 2, characterized in that, The update formula for the prototype center is expressed by the following formula: ； in, For the present The prototype center, For the updated The prototype center, Category in the current batch The sample set, The momentum coefficient, For the first The fusion feature vector of each sample.

4. The method according to claim 1, characterized in that, The cross-modal feature extraction network model includes a macroscopic branch and a microscopic branch that operate in parallel. The macroscopic branch is used to process the visible light image and the near-infrared image through non-local dilated convolutional blocks to extract macroscopic morphological features. The microscopic branch is used to process the microscopic image through a periodic-sensitive attention module to extract periodic microscopic structural features.

5. The method according to claim 1, characterized in that, The modal decoupling and gated fusion processing of the multi-scale feature vectors to obtain fused feature vectors includes: Apply orthogonal projection loss to the multi-scale feature vectors to obtain the decoupled multi-scale features; The decoupled multi-scale features are fused based on the gated cross-attention model to obtain the fused feature vector.

6. The method according to claim 1, characterized in that, The prototype contrast loss function is optimized using the following formula for intra-class to inter-class distance: ； in, For the first The fused feature vector of each sample, For the first The fused feature vector of each sample, For cosine similarity, As the prototype center of the category, For a set of samples of the same type that are difficult to bear, This is the temperature coefficient.

7. A deep learning-based image recognition system for traditional Chinese medicinal materials, characterized in that, The system includes: The image cross-modal alignment module is used to acquire multimodal images of Chinese medicinal materials, perform registration and alignment processing on the multimodal images, and obtain a cross-modal aligned image set. The multi-scale feature vector module is used to generate multi-scale feature vectors based on the cross-modal aligned image set and the cross-modal feature extraction network model. The feature fusion module is used to perform modal decoupling and gated fusion processing on the multi-scale feature vectors to obtain fused feature vectors. The Chinese medicinal material category output module is used to perform fine-grained classification based on the fused feature vector and the prototype contrast loss function, and output the Chinese medicinal material category. The image cross-modal alignment module is used to acquire visible light images, near-infrared images, and microscopic images of Chinese medicinal materials to form a multimodal image set; The multimodal image set is subjected to affine transformation-wavelet registration coupling processing to generate a cross-modal aligned image set; the error function in the affine transformation-wavelet registration is expressed by the following formula: ； in, For the image to be registered, For reference image, For Haar wavelet decomposition, Denotes the affine transformation matrix. The coefficients of the total variation regularization term, This is the smoothing constraint term for the transformation matrix.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images