A clip-guided multimodal fusion microscopic identification method

Abstract

The application relates to a CLIP-guided multi-modal fusion microscopic identification method, and relates to the field of microscopic image analysis, and aims to solve the problem of difficult microscopic identification of the component characteristics of Chinese patent medicines. Microscopic identification is used for true and false identification and quality evaluation by identifying the microstructure characteristics of medicinal materials, and traditional methods rely on manual analysis, are low in efficiency and strong in subjectivity. Deep learning improves the automatic performance, but still has problems such as dependence on labeled data, inaccurate feature positioning, large domain difference and insufficient use of professional knowledge, and increases the risk of misjudgment. Therefore, the application provides a CLIP-guided multi-modal fusion microscopic identification method for medicinal plant classification. Experiments show that the method can accurately position the key feature area of a microscopic image under the condition of no labeled data, can strengthen the feature extraction capability through professional knowledge in the pharmacopoeia, and can make the features have both fine-grained local texture information and associated information containing the position relationship of identification points through a dynamic feature fusion module. The application is suitable for true and false identification and quality detection of Chinese patent medicines.

Description

TECHNICAL FIELD

[0001] The present application is designed for microscopic identification of medicinal plants BACKGROUND

[0002] Microscopic identification of medicinal plants is a key means of identifying and evaluating the quality of traditional Chinese medicinal materials, which can provide scientific basis for medicinal material identification through microscopic feature analysis in microscopic images. However, the traditional microscopic identification method relies on manual operation and expert experience, which has the disadvantages of low efficiency and strong subjectivity, and is difficult to meet the actual needs of large-scale medicinal material detection. Therefore, the research on microscopic feature identification method based on automation technology has become the research focus in recent years.

[0003] The existing microscopic image identification method mainly relies on deep learning model to realize automatic processing through feature extraction and classification, although it improves the analysis efficiency to a certain extent, but still faces the following difficulties: first, the professional knowledge is not fully utilized; second, there is domain difference between standard microscopic feature image and scanning image; third, the labeled data is scarce and the acquisition cost is high, which further limits the performance of deep learning model.

[0004] In view of the above problems, a CLIP guided multi-modal fusion microscopic identification method is proposed, including a microscopic feature positioning module and a fine-grained classification module. In the microscopic feature positioning module, a self-loop prompt target detection network is proposed, which generates an initial pseudo mask of the microscopic image by using the SAM model, and combines a multi-level similarity feature contour de-duplication method to accurately locate the key area in the microscopic image under the condition of no labeled data; in the fine-grained classification module, a multi-modal feature fusion network is constructed, which extracts fine-grained local features of microscopic features through multi-layer convolution module, models the semantic features of pharmacopoeia text by using CLIP model to extract discriminant point position correlation features, and uses dynamic feature fusion module to deeply fuse the features, so as to realize accurate classification of microscopic images.

[0005] The present application breaks through the technical bottleneck of traditional microscopic image analysis method in positioning and classification by combining self-supervised learning and multi-modal contrast learning, effectively improves the positioning ability of key area of microscopic image under the condition of no labeled data, and provides an efficient technical solution for the true and false identification and quality detection of medicinal plants. SUMMARY

[0006] The purpose of the present application is to solve the misjudgment problem caused by the unclear feature division in the process of Chinese patent medicine component identification, and a CLIP guided multi-modal fusion microscopic identification method is proposed.

[0007] The above invention purpose is mainly realized by the following technical scheme:

[0008] S1. Use a slice scanner to scan the traditional Chinese medicine sample to obtain a panoramic image. Then, convert the panoramic image into a common format image and use a fixed-size window to cut the image into smaller images.

[0009] S2. Obtaining the location information of microscopic features of traditional Chinese medicine using a self-looping cueing target detection network: First, an initial pseudo-mask for the microscopic features is generated using the SAM model. Then, the mask is refined using the contour deduplication module of the multi-level similarity features of the self-looping cueing target detection network. The refined mask is then used as cue information to generate the mask for the next loop, realizing iterative optimization of the pseudo-mask so as to accurately locate the microscopic features of traditional Chinese medicine without labeled data. The steps are as follows:

[0010] (1) Use the SAM model to generate the initial pseudomask of the microscopic image to provide preliminary location information for the target area in the microscopic image;

[0011] (2) Extract the contour from the binarized mask in the microscopic image and select the contour with the largest area as the main contour;

[0012] (3) Based on the Fourier descriptor, calculate the global shape similarity between the main contour and the stored contours. If the Euclidean distance is less than a preset threshold, the similarity is calculated. D If the shape is similar globally, then it is determined that the shape is similar, and the calculation formula is as follows:

[0013]

[0014] d D =||D new -D stored ||<∈ D (2)

[0015] In the formula D represents the Fourier transform coefficients. k This is the normalized descriptor.

[0016] (4) Based on the Hu moment feature, calculate the local detail similarity between the main contour and the stored contour. If the Euclidean distance is less than a preset threshold, the similarity is calculated. H If the result is similar in local details, it is determined to be similar in local details. The calculation formula is as follows:

[0017] H i =∑ p+q＝i μ pq (i=1,…,7) (3)

[0018] d H =||H new -H stored ||<∈ H (4)

[0019] In the formula μpq Hu moment of the image, H i Hu moment is a rotation invariant.

[0020] (5) Based on the contour area ratio, the scale consistency of the main contour and the stored contour is judged. If the area ratio is greater than the preset threshold A , it is considered that the scale is consistent, and the calculation formula is as follows:

[0021]

[0022] If the above three conditions are met, the current main contour is marked as repeated and filtered; if any condition is not met, the current main contour is added to the effective mask set.

[0023] (6) The position information screened through steps (3)-(5) is used as the input prompt for the next round, and the feature region of the pseudo mask is gradually optimized;

[0024] (7) When the Dice loss in the loop iteration does not decrease significantly for five rounds in a row, output the final optimized feature region position;

[0025] S3, extract fine-grained local features and position correlation features of Chinese herbal medicine microscopic features: take the feature region of the microscopic image as the input, extract the features through multi-layer convolution, and combine the text texture description in the pharmacopoeia. Through the CLIP model, the microscopic image and the pharmacopoeia text information are fused, so that the image features can contain relevant information between identification points:

[0026] (1) The feature region of the microscopic image is processed by a feature extraction module, which includes an MBConv multi-layer convolution module and a CLIP module;

[0027] (2) The multi-layer convolution module extracts fine-grained local features of the feature region through deep convolution;

[0028] (3) In the CLIP module, the feature vector of the text description in the pharmacopoeia is extracted, which is mapped to the same semantic space as the extracted image features. Through the shared semantic space, the image features have semantic information about the position relationship of the identification points;

[0029] (4) The fine-grained local features and identification point position correlation features generated by the multi-layer convolution module and the CLIP module are input into the dynamic feature fusion module for feature fusion;

[0030] S4, construct a dynamic feature fusion module: group process the features of the multi-layer convolution module and the CLIP module through a multi-head attention mechanism, and use an adaptive gating mechanism to weight and optimize the importance of features in different channels;

[0031] (1) In the multi-head channel attention mechanism, the input fusion features are divided into multiple subspaces, and the position correlation features of each channel are extracted through global context information. The calculation formula of the position correlation features is as follows:

[0032]

[0033] wherein represents a global average pooling operation, is the input channel feature, c i represents the extracted position correlation information, which is used to describe the position relationship between the discriminant points in the channel.

[0034] (2) Based on the position correlation information, the attention weights of the channel are generated through multi-layer linear transformation and activation function, which are used to adjust the importance of the channel features. The calculation formula of the attention weights is as follows:

[0035]

[0036] wherein σ is a Sigmoid activation function, δ is a ReLU activation function, W 1,i ,W 2,i is a linear transformation matrix, b 1,i and b 2,i are bias terms.

[0037] (3) The input features are weighted through the channel attention weights, which are used to strengthen the key features and suppress the redundant information. The calculation formula is as follows:

[0038]

[0039] wherein ⊙ represents a channel-wise weighting operation, which ensures that the optimized features contain more task-related information.

[0040] (4) In the adaptive gating mechanism, the gating factor is generated through the channel normalization operation. The calculation formula of the gating factor is as follows:

[0041]

[0042] wherein IN(·) represents a channel normalization function, σ is a Sigmoid activation function, and β is a gating factor, which is used to dynamically adjust the proportion of local detail features and position correlation features.

[0043] (5) The optimized features are dynamically weighted through the gating factor. The calculation formula is as follows:

[0044]

[0045] wherein ⊙ represents a channel-wise weighting operation.

[0046] S5, calculating a loss function: the CLIP loss calculates the similarity of image features and text features, the classification loss calculates the difference between model output and real labels, and the final loss is composed of the CLIP loss and the classification loss to optimize model parameters;

[0047] (1) The CLIP loss calculates the similarity of image features and text features to strengthen the interaction of semantic information and image information, and the formula is as follows:

[0048]

[0049] In the formula, represents the calculation of cosine similarity, and are image features and text features respectively.

[0050] (2) The classification loss calculates the difference between model output and real labels, and the formula is as follows:

[0051]

[0052] In the formula, represents the fusion features of the i-th sample, is the cross-entropy loss function.

[0053] (3) The classification loss function and the CLIP similarity loss function are combined to optimize the classification performance of the model, and the total loss function calculation formula is as follows:

[0054]

[0055] In the formula, λ MB and λ clip are the gating scores corresponding to the fine-grained local features and the position correlation features.

[0056] Inventive effect

[0057] The application provides a CLIP-guided multi-modal fusion microscopic identification method. The method first generates an initial pseudo mask of a microscopic image using a SAM model, and iteratively optimizes the pseudo mask through a self-loop prompting target detection network to accurately locate the key feature area in the microscopic image; then extracts visual features of the feature area through a multi-layer convolution module, and extracts associated features containing the position relationship of identification points through a CLIP model combined with the textual identification standard in the pharmacopoeia; finally, a dynamic feature fusion module is constructed, which processes the visual features and associated features through a multi-head channel attention mechanism, and dynamically adjusts the importance of the feature channels through an adaptive gating mechanism to enhance the ability to capture fine-grained features; finally, the classification network obtains the final classification label, and completes the microscopic component identification task of traditional Chinese medicine. Experiments show that the application has the following advantages: (1) accurately locating the key feature area through an adaptive cyclic prompting mechanism improves the positioning ability under the condition of no labeled data; (2) aligning the visual features and textual features of the feature area through the CLIP model makes the image features contain the position relationship of identification points; (3) the dynamic feature fusion module cooperatively fuses the features, so that the final features include not only fine-grained local information but also associated information with the position relationship of identification points. The application is suitable for the authenticity identification and quality evaluation of medicinal plants, and provides an efficient microscopic image analysis method. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 CLIP-guided multi-modal fusion microscopic identification method flowchart;

[0059] Figure 2 Self-loop prompting target detection network structure diagram;

[0060] Figure 3 Multi-modal fusion classification network structure diagram;

[0061] Figure 4 Dynamic feature fusion module structure diagram;

[0062] Figure 5 Traditional Chinese medicine microscopic feature identification effect diagram; DETAILED DESCRIPTION Specific implementation method one:

[0064] To make the objectives, technical solutions, and advantages of the embodiments of the application clearer, the technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are some, but not all, of the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the application.

[0065] As Figure 1 shown, the CLIP guided multi-modal fusion microscopic identification method includes three modules of data processing, microscopic feature positioning and fine-grained classification. The data processing module mainly divides the scanning image into small images, the microscopic feature positioning module mainly locates the traditional Chinese medicine feature area through the self-loop prompt target detection network, and the fine-grained classification module classifies the feature area through the CLIP guided fine-grained classification network. The specific steps are as follows:

[0066] S1, scanning the traditional Chinese medicine sample by using a slice scanner to obtain a panoramic image, then converting the panoramic image into a common format image, and then dividing the image into small images by using a fixed size window;

[0067] S2, using a self-loop prompt target detection network to obtain the position information of the microscopic features of Chinese herbal medicine: first, using a SAM model to generate an initial pseudo mask of the microscopic features, then using a multi-level similarity feature contour de-duplication module of the self-loop prompt target detection network to finely screen the mask, and using the finely screened mask as prompt information to generate a mask for the next cycle, so as to iteratively optimize the pseudo mask, so as to accurately locate the microscopic features of Chinese herbal medicine under the condition of no labeled data;

[0068] S3, extracting fine-grained local features and position correlation features of the microscopic features of Chinese herbal medicine: taking the feature area of the microscopic image as input, extracting features through multi-layer convolution, and combining the text texture description in the pharmacopoeia, and fusing the microscopic image and the pharmacopoeia text information through the CLIP model, so that the image features can be rich in correlation information between identification points;

[0069] S4, constructing a dynamic feature fusion module: the dynamic feature fusion module processes the features from the feature extraction module and the CLIP model through a multi-head attention mechanism, and uses an adaptive gating mechanism to weight and optimize the importance of features in different channels;

[0070] S5, model training and optimization: the CLIP loss calculates the similarity of the image features and the text features, the classification loss calculates the difference between the model output and the real label, and the sum of the CLIP loss and the classification loss constitutes the final loss to optimize the model parameters;

[0071] S6, result prediction: input the microscopic features of Chinese herbal medicine to be classified into the classification network, after feedforward propagation, the classification network calculates the probability of each class, and finally outputs the class label corresponding to the image;

[0072] The embodiments of the present application will be described in detail as follows:

[0073] The embodiments of the present application use 10 scanning images of medicinal plant slices, and the identification and analysis are realized by using the algorithm of the present application, which is specifically realized as follows.

[0074] S1, scanning the traditional Chinese medicine sample by using a slice scanner to obtain a panoramic image, then converting the panoramic image into a common format image, and then cutting the image into small images by using a fixed size window;

[0075] As shown in Figure 2 , the self-loop prompting target detection network training includes the following steps:

[0076] S2, using a self-loop prompting target detection network to obtain the position information of the microscopic features of Chinese herbal medicine: first, using a SAM model to generate an initial pseudo mask of the microscopic features, then using a multi-level similarity feature contour de-duplication module of the self-loop prompting target detection network to fine screen the mask, and using the fine screened mask as the prompt information to generate the mask of the next cycle, realizing the iterative optimization of the pseudo mask, so as to accurately locate the microscopic features of Chinese herbal medicine under the condition of no labeled data, and the steps are as follows:

[0077] (1) inputting an image, and extracting a feature map by using an image encoder of a SAM model;

[0078] (2) initial pseudo mask generation, the mask decoder decodes the features to generate an initial pseudo mask. The initial pseudo mask generation formula is as follows:

[0079] M0=σ(Decoder(F prompt )),M0∈[0,1] H×W (14)

[0080] In the formula, sigma is a Sigmoid activation function, and the output M0 is an initial pseudo mask.

[0081] (3) the initial pseudo mask is processed by binarization to divide the pixel value into a target region or a background region. Then, the contour of the target region is extracted for subsequent screening and optimization;

[0082] (5) the contour of the target region is screened and de-duplicated by using Fourier descriptors, Hu moments and area ratios, and only the most relevant regions are retained;

[0083] The low frequency component of the Fourier descriptor is used to describe the overall shape feature of the contour, which can effectively distinguish the targets with similar shapes but different sizes or positions, and the calculation formula is as follows:

[0084]

[0085] In the formula, c n is a complex representation of the contour point, and N is the number of contour points.

[0086] To capture the local detail features of the contour, the Hu moment based on the center moment of the image is introduced. The Hu moment can effectively represent the local geometric features of the contour, and its calculation formula is:

[0087]

[0088] In the formula, μ pq represents the center moment of the image.

[0089] Finally, the area ratio is used as the judgment basis, and by calculating the ratio of the target region contour area to the background area, it is further judged whether the target region meets the scale consistency requirement of the expected region.

[0090] Combined with the above multi-level features, the comparison and screening of the contour features are only retained for the target region that meets all the above conditions, so as to obtain a more accurate microscopic image target region.

[0091] (6) The mask position information after the current round of fine screening is input into the SAM model as prompt information to generate the next round of mask, and the Dice loss is used to evaluate the mask quality to optimize the target region positioning;

[0092] First, the boundary prompt box B t is extracted from the binary mask generated in the current round .

[0093] Next, the prompt box B t and the input image I are passed to the SAM model to generate a new round of mask M t+1 , and the process is as follows:

[0094] M t+1 = SAM (I, Bt) (17)

[0095] In order to evaluate the optimization effect of each round, the Dice loss function is introduced to measure the matching degree of the generated mask M t+1 and the label M true , and the formula is:

[0096]

[0097] In the formula, the numerator is the intersection of the generated mask and the true label, and the denominator is the sum of the two. The smaller the Dice loss is, the closer the generated mask is to the target region. The whole optimization is carried out in a loop until the Dice loss no longer changes significantly in the last 5 iterations, at which time it is considered that the model has converged, and the final optimized mask M * = M T is output.

[0098] As shown in the training of the fine-grained classification network Figure 3 , the steps include:

[0099] S3. Extracting fine-grained local features and positional correlation features from the microscopic features of traditional Chinese medicine: Using the feature regions of the microscopic image as input, feature extraction is performed through multi-layer convolution. Combined with the text texture description in the pharmacopoeia, the microscopic image and pharmacopoeia text information are fused using the CLIP model, so that the image features are rich in the correlation information between identification points. The steps are as follows:

[0100] (1) Fine-grained local features of the feature region are extracted through a multi-layer convolution module, as shown in the following formula:

[0101] Y = X + ReLU6(Conv) 1x1 (DepthwiseConv(Conv 1x1 (X)))) (19)

[0102] In the formula, X is the input feature map, and Conv 1x1 The function is a 1x1 convolution, DepthwiseConv is a depthwise convolution, and ReLU6 is the activation function. ReLU6 limits the output range to [0,6] to avoid overactivation.

[0103] (2) Visual features of the microscopic image are extracted using the CLIP model image encoder. Let I be the input microscopic image, then the extracted visual features F img It can be represented as:

[0104] F img =ImageEncoder(I) (20)

[0105] (3) Input the textual descriptions of microscopic identification of medicinal plants from the pharmacopoeia into the text encoder of the CLIP model to extract semantic features of the microscopic images. Let T be the textual description of the microscopic structure in the pharmacopoeia. Then the extracted text embedding can be represented as:

[0106] F text =TextEncoder(T) (21)

[0107] (4) Visual features F img and semantic features F text They are embedded into the same multimodal space, and multimodal feature alignment is performed through a contrastive learning mechanism. The formula is shown below:

[0108]

[0109] In the formula, cos() represents the cosine similarity, used to measure the similarity between image and text embeddings. N is the number of samples in the training batch.

[0110] like Figure 4 The dynamic feature fusion module shown includes the following steps:

[0111] S4, constructing a dynamic feature fusion module: the dynamic feature fusion module processes the features from the feature extraction module and the CLIP model through a multi-head attention mechanism, and uses an adaptive gating mechanism to weight and optimize the importance of features in different channels, and the steps are as follows:

[0112] (1) The position correlation information of each feature channel is extracted by the global average pooling operation, and the calculation formula is as follows:

[0113]

[0114] wherein is the input channel feature, H and W are the height and width of the feature map respectively, c i represents the position correlation information of the channel.

[0115] (2) The attention module is used to generate channel weight α i , which is used to adjust the importance of the channel feature, and the specific formula is as follows:

[0116] α i =σ(W2·δ(W1·c i +b1)+b2) (24)

[0117] wherein W1 and W2 are linear transformation matrices, b1 and b2 are bias terms, δ is a ReLU activation function, and σ is a Sigmoid activation function.

[0118] (3) The input feature is weighted by the attention weight α i of each channel to generate the feature The calculation formula is as follows:

[0119]

[0120] wherein ⊙ represents the element-wise multiplication operation.

[0121] (4) The weighted feature channel is input into the adaptive gating mechanism, which dynamically adjusts the importance of the feature channel combined with the local context information. The gating factor β is generated, and the calculation formula is as follows:

[0122] β=σ(W gate ·δ(IN(x f ))) (26)

[0123] wherein IN() is a normalization operation, W gate is the weight matrix of the gating mechanism, δ is a ReLU activation function, and σ is a Sigmoid activation function.

[0124] (5) The optimized features are dynamically weighted using a gating factor to further balance the importance of the detail feature and the position-related feature, and the calculation formula is as follows:

[0125]

[0126] In the formula, represents a channel-by-channel weighting operation.

[0127] S5, model training and optimization: the CLIP loss calculates the similarity of the image feature and the text feature, the classification loss calculates the difference between the model output and the real label, and the sum of the CLIP loss and the classification loss constitutes the final loss to optimize the model parameters, and the steps are as follows:

[0128] (1) The CLIP loss calculates the similarity of the image feature and the text feature, and the formula is as follows:

[0129]

[0130] In the formula, represents the cosine similarity calculation, and are the image feature and the text feature respectively.

[0131] (2) The classification loss calculates the difference between the model output and the real label, and the formula is as follows:

[0132]

[0133] In the formula, represents the fusion feature of the i-th sample, is the cross-entropy loss function.

[0134] (3) The classification performance of the model is optimized by combining the classification loss function and the CLIP similarity loss function , and the total loss function calculation formula is as follows:

[0135]

[0136] In the formula, λ MB and λ clip are the gating scores corresponding to the fine-grained local feature and the position-related feature.

[0137] S6, result prediction: the micro features of the Chinese herbal medicine to be classified are input into the classification network, and after feedforward propagation, the classification network calculates the probability of each class, and finally outputs the class label corresponding to the image;

[0138] The final implementation effect is shown in Figure 5 , and it can be seen from the figure that the feature area can be accurately located and the type of the feature can be accurately identified.

[0139] The application also can have other various embodiments, and those skilled in the art can make various corresponding changes and modifications according to the application without departing from the spirit and essence of the application, and these corresponding changes and modifications should all belong to the scope of the application.

Claims

1. A CLIP-guided multimodal fusion microscopy method, characterized in that, The method comprises the following steps: S1, scanning the traditional Chinese medicine sample by using a slice scanner to obtain a panoramic image, then converting the panoramic image into a common format image, and then cutting the image into small images by using a fixed size window; S2, obtaining the position information of the Chinese herbal medicine microscopic features by using a self-loop prompting target detection network: first, using a SAM model to generate an initial pseudo mask of the microscopic features, then using a multi-level similarity feature contour de-duplication module of the self-loop prompting target detection network to fine-screen the mask, and using the fine-screened mask as the prompt information to generate the mask of the next cycle, realizing the iterative optimization of the pseudo mask, so as to accurately locate the Chinese herbal medicine microscopic features under the condition of no labeled data; S3, extracting the fine-grained local features and position correlation features of the Chinese herbal medicine microscopic features: taking the feature area of the microscopic image as the input, extracting the features by multi-layer convolution, and combining the text texture description in the pharmacopoeia, and fusing the microscopic image and the pharmacopoeia text information by the CLIP model, so that the image features contain the correlation information between the identification points; S4, constructing a dynamic feature fusion module: grouping the features from the feature extraction module and the CLIP model by using a multi-head attention mechanism, and using an adaptive gating mechanism to weight and optimize the features of different channels; S5, model training and optimization: the CLIP loss calculates the similarity of the image features and the text features, the classification loss calculates the difference between the model output and the real label, and the sum of the CLIP loss and the classification loss constitutes the final loss to optimize the model parameters; S6, result prediction: inputting the Chinese herbal medicine microscopic features to be classified into the classification network, after feedforward propagation, the classification network calculates the probability of each category, and finally outputs the category label corresponding to the feature.

2. The CLIP-guided multimodal fusion microscopic identification method of claim 1, wherein, The step S2 of obtaining the position information of the Chinese herbal medicine microscopic features by using the self-loop prompting target detection network comprises the following steps: S21, generating an initial pseudo mask of the microscopic image by using a SAM model to provide preliminary position information for the target area in the microscopic image; S22, introducing a self-loop prompting target detection network, using the position information generated in the current round as the input prompt in the next round, and gradually optimizing the boundaries and feature areas of the pseudo mask; S23, in the process of loop prompting, using a multi-level similarity feature contour de-duplication method to screen and optimize the repeated masks, so as to ensure that each target area is labeled only once; S24, when the Dice loss in the loop iteration has no significant improvement for five consecutive rounds, outputting the final optimized Chinese herbal medicine microscopic feature area mask.

3. The CLIP-guided multimodal fusion microscopic identification method of claim 1, wherein, The multi-level similarity feature contour de-duplication method in step S2 comprises the following steps: S25, extracting the contour of the binary mask of the Chinese herbal medicine feature area, and selecting the largest contour as the main contour; S26, extracting the multi-level similarity features of the main contour, including Fourier descriptors, Hu moment features and contour area ratios; S27、Based on the Fourier descriptor, the global shape similarity between the main profile and the stored profile is calculated, and if the Euclidean distance is less than a preset threshold ∈ D , it is judged that the global shape is similar, and the calculation formula is as follows: d D =||D new -D stored ||<∈ D (2) wherein are Fourier transform coefficients, D k is a normalized descriptor; S28, based on the Hu moment feature, calculate the local detail similarity of the main contour and the stored contour, if the Euclidean distance is less than the preset threshold H , judge as local detail similarity, the calculation formula is as follows: H i =∑ p+q＝i μ pq (i=1,...,7) (3) d H =||H new -H stored ||<∈ H (4) where μ pq denotes the central moments of the image, H = (H1, H2,..., H7), H i denotes the rotation invariants of the Hu moments; S29, judging the scale consistency of the new contour and the stored contour based on the contour area ratio, if the area ratio is greater than a preset threshold ∈ A then considering the scale consistency, and the calculation formula is as follows: If all the three conditions are met, the current contour is marked as a repetition and filtered; if any condition is not met, the current main contour is added to the valid mask set.

4. The CLIP-guided multimodal fusion microscopic identification method of claim 1, wherein, In step S3, the fine-grained local features and position correlation features of the Chinese herbal medicine microscopic features are extracted: taking the feature region of the microscopic image as input, the feature extraction module is used for feature extraction, and the CLIP model is used for fusion of the microscopic image and the text information in the pharmacopoeia, so that the image features can be rich in correlation information between identification points. The following steps are included: S31, processing the Chinese herbal medicine microscopic features through a feature extraction module, wherein the feature extraction module comprises a multi-layer convolution module and a CLIP module; S32, the multi-layer convolution module extracts fine-grained local features of the image through deep convolution; S33, in the CLIP module, the feature vector of the text description in the pharmacopoeia is extracted, which is mapped to the same semantic space as the extracted image features, and the image features are made to have global correlation information containing the position relationship of the identification points through the shared semantic space; S34, the features generated by the multi-layer convolution module and the CLIP module are input into a dynamic feature fusion module for collaborative fusion.

5. The dynamic integrated classification method for microscopical identification of medicinal plants as claimed in claim 1 wherein, In step S4, the dynamic feature fusion module is constructed: the features from the feature extraction module and the CLIP model are processed in groups through a multi-head attention mechanism, and the importance of the features in different channels is weighted using an adaptive gating mechanism, which has the following steps: S41, in the multi-head channel attention mechanism, the input fusion features are divided into multiple subspaces, and the context correlation features of each channel are extracted through the identification point position correlation information, and the calculation formula of the context correlation features is as follows: In the formula represents a global average pooling operation, is the input channel feature, c i represents the extracted position correlation information; S42, based on the extracted position correlation information, the attention weights of the channels are generated through multi-layer linear transformation and activation function to adjust the importance of the channel features, and the calculation formula is as follows: where σ is a Sigmoid activation function, δ is a ReLU activation function, W 1,i ,W 2,i is a linear transformation matrix, b 1,i and b 2,i are bias terms; S43, the input features are weighted using the channel attention weights to strengthen the key features and suppress the redundant information, and the calculation formula is as follows: Wherein, ⊙ represents the channel-by-channel weighting operation; S44, in the adaptive gating mechanism, the channel normalization operation is used to generate the gating factor, and the calculation formula is as follows: Wherein, IN(·) represents the channel normalization function, σ is the Sigmoid activation function, and β is the gating factor; S45, the optimized features are dynamically weighted using the gating factor to further balance the importance of the detail features and the position correlation features, and the calculation formula is as follows: Wherein, ⊙ represents the channel-by-channel weighting operation.

6. The dynamic integrated classification method for microscopical identification of medicinal plants as claimed in claim 1 wherein, Step S5 calculates the loss function: the CLIP loss calculates the similarity between the image features and the text features, the classification loss calculates the difference between the model output and the real label, and the final loss is composed of the CLIP loss and the classification loss to optimize the model parameters; S51, the CLIP loss calculates the similarity between the image features and the text features to strengthen the interaction between the semantic information and the image information, and the formula is as follows: In the formula denotes the cosine similarity calculation, and are the feature representations of the image and text, respectively; S52, the classification loss calculates the difference between the model output and the real label, and the formula is as follows: In the formula denotes the fusion feature of the i-th sample, is a cross-entropy loss function; S53, combine the classification loss function and the CLIP similarity loss function The total loss function of the optimization model is calculated as follows: where λ MB and λ clip are gating scores corresponding to fine-grained local features and location-dependent features, respectively.

Images